You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Nick Wellnhofer <we...@aevum.de> on 2016/05/15 18:38:17 UTC

[lucy-dev] Separate binaries for test suites

Lucifers,

I'd like to tackle an issue that's been nagging me for quite some time and 
build separate binaries for our test suites. Then the test binaries can be 
skipped when installing a Clownfish project.

This isn't too hard for the C bindings, but for the Perl bindings, it's more 
complicated. My current plan is to move the test code into its own directory 
separate from "core". Then I'd like to add a mechanism to create distinct 
dynamic libraries (XS "modules") for each source directory. The build system 
gets a specification like:

     xs_modules => [
         {
             name => 'Clownfish',
             source_dirs => [ 'core' ],
         },
         {
             name => 'Clownfish::Test',
             source_dirs => [ 'tests' ],
         },
     ]

Then separate .xs, boot.h, boot.c, and callbacks.c files are generated for 
each module, and linked with the .c files in each source directory.

As a first step, I'd move all the test code into its own directory, simply 
named "tests".

What do you think?

Nick

Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On 24/05/2016 04:33, Marvin Humphrey wrote:
>> These compiler flags must be reused when compiling code for host language
>> bindings. A good example is `-m32` to create 32-bit binaries on 64-bit
>> systems. Other examples are '-mms-bitfields` (used by some Windows Perls)
>> and flags concerning data alignment.
>>
>> So we'd have scan the host language C flags and make sure to reuse all flags
>> that cause incompatible code to be generated. Since there is no conclusive
>> list of all these flags, this approach is error-prone and will probably need
>> occasional updates. I'm not saying it can't be done, but for now, I'd prefer
>> to simply use all the compiler flags proposed by the host language. Maybe it
>> would make more sense to have a blacklist of flags that are safe to remove?
>
> My guess, which admittedly doesn't have a lot to back it up, is that although
> there is an open set of potentially incompatible flags, there aren't that many
> that we wouldn't find out about right away and be able to fix via
> whitelisting.

Probably. My only concern is that some flags might result in subtle 
incompatibilities that don't cause problems with our current code base, but 
might do so later. Such cases will be extremely hard to debug.

>> I think adding support for multiple parcels with separate binaries per
>> parcel isn't much harder. Instead of building a binary with test and
>> non-test code, we'd build a binary that contains only the test code. This
>> should have other uses besides testing. OTOH, having separate binaries
>> places some restrictions on the test code.
>
> That may be, but as discussed before,
> (<http://markmail.org/message/tmn4jb3yveftumu5>) Clownfish parcels were
> intended to represent atomic units of installation in order to facilitate
> versioning of dependencies.  Having multiple parcels installed together
> complicates the mechanisms to handle resolving compatible versions and advise
> what and when to upgrade.

My plan is to create separate .xs and .so files for every parcel, each having 
their own version. So in theory, every parcel could also be installed separately.

Nick


Re: [lucy-dev] Separate binaries for test suites

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Jun 14, 2016 at 5:28 AM, Nick Wellnhofer <we...@aevum.de> wrote:
> On Jun 8, 2016, at 03:02 , Marvin Humphrey <ma...@rectangular.com> wrote:
>>
>> Over time, we should try to make decisions that consolidate as much of build
>> code as possible in a shared central system (i.e. charmonizer/make for the
>> foreseeable future).  Even if that code eventually gets refactored, it's only
>> one task to refactor it, as opposed to N tasks to refactor code in N hosts.
>
> Originally, I didn’t want to make Charmonizer a hard requirement but I’m
> willing to give that up, at least for now.

I hear you.  As Charmonizer has expanded a bit, I've been reminded of the old
Larry Wall chestnut, "It's easier to port a shell than a shell script."

What's more important than Charmonizer's implementation are satisfying the
requirements of not imposing any prerequisites (e.g. Unix shell) to run
configuration probing of a C compilation environment, nor any build tool
outside of what the host language environment provides.  But I figure that
means for the time being, it's impractical to work with anything except
bundle-able, portable Charmonizer.

>> I would say tests in a separate binary.  For example, under the Perl bindings
>> the core tests for Lucy would be run from a separate XS module (Lucy::CFTest?
>> LucyCFTest? CFTest?) which has a dependency on the main Lucy XS module.
>
> Maybe we should keep the Lucy::Test convention?

After a bit of reflection: +1

I think that means also that *::Test is reserved for any Clownfish parcel.

Marvin Humphrey

Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On Jun 8, 2016, at 03:02 , Marvin Humphrey <ma...@rectangular.com> wrote:
> 
> Over time, we should try to make decisions that consolidate as much of build
> code as possible in a shared central system (i.e. charmonizer/make for the
> foreseeable future).  Even if that code eventually gets refactored, it's only
> one task to refactor it, as opposed to N tasks to refactor code in N hosts.

Originally, I didn’t want to make Charmonizer a hard requirement but I’m willing to give that up, at least for now.

> I would say tests in a separate binary.  For example, under the Perl bindings
> the core tests for Lucy would be run from a separate XS module (Lucy::CFTest?
> LucyCFTest? CFTest?) which has a dependency on the main Lucy XS module.

Maybe we should keep the Lucy::Test convention?

> That's a solvable problem: we just rename such dirs when moving them to the
> host language distro.  Then we can find the dirs using logic like this:
> 
>    my $CFCORE = -d "cfcore" ? "cfcore" : "../core";
>    my $CFTEST = -d "cftest" ? "cftest" : "../test”;

OK, then the next step is to move the test code.

Nick


Re: [lucy-dev] Separate binaries for test suites

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Sat, Jun 4, 2016 at 12:32 PM, Nick Wellnhofer <we...@aevum.de> wrote:
> On 01/06/2016 17:04, Nick Wellnhofer wrote:
>>
>> On 26/05/2016 04:12, Marvin Humphrey wrote:
>>>
>>> With Python distutils as with Perl's Module::Build, it is at least
>>> theoretically possible to provide a list of object files to be linked
>>> into the
>>> host extension.
>>
>>
>> I'm leaning towards the "list of object files" approach for the Perl
>> bindings.  Unless static libraries are really used as intended, they seem
>> to create more problems than they solve.
>
> I submitted a new pull request the makes the Perl bindings use "make" to
> build the core object files.

+1 to merge!

Over time, we should try to make decisions that consolidate as much of build
code as possible in a shared central system (i.e. charmonizer/make for the
foreseeable future).  Even if that code eventually gets refactored, it's only
one task to refactor it, as opposed to N tasks to refactor code in N hosts.

This branch has some complexities, but it moves us in the right direction.

> Do we want the tests in a separate binary, or do we want a second binary
> containing both core and test code? The latter allows to integrate the test
> code more tightly, the former requires to make some test-only symbols
> visible. An advanced solution would even allow both approaches.

I would say tests in a separate binary.  For example, under the Perl bindings
the core tests for Lucy would be run from a separate XS module (Lucy::CFTest?
LucyCFTest? CFTest?) which has a dependency on the main Lucy XS module.

Most unit tests operate against the public API.  In my opinion, it is OK for
test code to be written as if it were external to the library.

It may be necessary for the test-author to modify the public API in order to
make some kinds of testing possible, or pursue other workarounds in unusual
cases.  That's acceptable.

Some test code will need access to struct definitions.  Since the tests will
only ever operate against the current version, we don't need to be concerned
about ABI compatibility promises.  It should continue to be possible to access
struct definitions, but it would be nice if the interface were a little less
obscure than the current mechanism of defining a `C_PREFIX_CLASSNAME` macro.

> Currently, we pass parcel privacy defines like CFP_CFISH as command-line
> arguments to the compiler. If we start to build separate binaries, this
> requires to put the test code in a separate directory and use special
> Makefile rules that add some per-directory flags. I think this can be done
> in a cross-platform way but it's a little complicated. Another approach is
> to move the #defines directly into the source files. This is more flexible
> with regard to directory layout, but less flexible when changing
> configurations. Different host languages might even need different settings.

I think "convention over configuration" is a good operating principle here.
Supporting flexibility in directory layouts would be a misfeature.

Complicated build code is technical debt and a barrier to contribution.  It is
often appropriate to take on some debt when evolving a codebase quickly and
when final solutions are not clear, but hopefully we can pay down that debt
eventually.

> It's probably a good idea to move the test code into a separate directory
> regardless of the issue above. Marvin suggested to name it "test". The only
> problem I can see is that this directory must be moved into the host
> language subdirectory when bundling host language distributions like CPAN
> tarballs. "test" is a pretty generic name that might cause clashes, but the
> same is true for "core".

That's a solvable problem: we just rename such dirs when moving them to the
host language distro.  Then we can find the dirs using logic like this:

    my $CFCORE = -d "cfcore" ? "cfcore" : "../core";
    my $CFTEST = -d "cftest" ? "cftest" : "../test";

Marvin Humphrey

Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On 01/06/2016 17:04, Nick Wellnhofer wrote:
> On 26/05/2016 04:12, Marvin Humphrey wrote:
>> With Python distutils as with Perl's Module::Build, it is at least
>> theoretically possible to provide a list of object files to be linked into the
>> host extension.
>
> I'm leaning towards the "list of object files" approach for the Perl bindings.
> Unless static libraries are really used as intended, they seem to create more
> problems than they solve.

I submitted a new pull request the makes the Perl bindings use "make" to build 
the core object files.

Some open questions:

Do we want the tests in a separate binary, or do we want a second binary 
containing both core and test code? The latter allows to integrate the test 
code more tightly, the former requires to make some test-only symbols visible. 
An advanced solution would even allow both approaches.

Currently, we pass parcel privacy defines like CFP_CFISH as command-line 
arguments to the compiler. If we start to build separate binaries, this 
requires to put the test code in a separate directory and use special Makefile 
rules that add some per-directory flags. I think this can be done in a 
cross-platform way but it's a little complicated. Another approach is to move 
the #defines directly into the source files. This is more flexible with regard 
to directory layout, but less flexible when changing configurations. Different 
host languages might even need different settings.

It's probably a good idea to move the test code into a separate directory 
regardless of the issue above. Marvin suggested to name it "test". The only 
problem I can see is that this directory must be moved into the host language 
subdirectory when bundling host language distributions like CPAN tarballs. 
"test" is a pretty generic name that might cause clashes, but the same is true 
for "core".

Nick


Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On 26/05/2016 04:12, Marvin Humphrey wrote:
> With Python distutils as with Perl's Module::Build, it is at least
> theoretically possible to provide a list of object files to be linked into the
> host extension.

I'm leaning towards the "list of object files" approach for the Perl bindings. 
Unless static libraries are really used as intended, they seem to create more 
problems than they solve.

> But with CGO, I don't know how to do that.
>
> *   You can't tell the Go build system about a list of compiled object files.
> *   You can't tell CGO about arbitrary directories full of C files to compile.
> *   You *can* put loose C files into the package directory, but only that one
>     dir -- nested dirs aren't allowed.
> *   You *can* hack in linker instructions using pseudo `#cgo` directives.
>     https://golang.org/cmd/cgo/#hdr-Using_cgo_with_the_go_command
>
> We're currently using the last option to tell CGO about the static archive
> built using Charmonizer/Make.

Another solution is to add fake references to object files that aren't picked 
up when linking with the static library. The only problematic file for me was 
TestUtils.o. Object files for non-inert classes are always referenced by the 
generated binding code. For inert classes, this could even be automated by 
adding a reference to any symbol of an inert class somewhere in the generated 
code. This leaves non-Clownfish source files (stuff like LFReg) that aren't 
used by the parcel itself which should be a rare case.

>> What about code like `xs/XSBind.c`? Should it be compiled by the host, too?
>
> Yes, IMO -- it's hard to handle otherwise because you have to know about
> installation-specific include dirs and the like.

OK.

Nick


Re: [lucy-dev] Separate binaries for test suites

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Wed, May 25, 2016 at 4:29 AM, Nick Wellnhofer <we...@aevum.de> wrote:

> There's another complication which might affect the Python and Go bindings,
> too. If you link a static library into a dynamic library with GNU binutils,
> you have to use the `--whole-archive` linker option. Otherwise, some object
> files in the static library might be ignored. So the link command should
> look something like
>
>     gcc \
>         [.o files] \
>         -Wl,--whole-archive \
>         libclownfish.a \
>         -Wl,--no-whole-archive \
>         [dynamic libs] \
>         ...
>
> The Solaris linker seems to require `-z allextract` and `-z defaultextract`.
> On OS X, we need `-all_load`. So it seems that we have to detect the linker
> as well.

Mmf. :(

With Python distutils as with Perl's Module::Build, it is at least
theoretically possible to provide a list of object files to be linked into the
host extension.

But with CGO, I don't know how to do that.

*   You can't tell the Go build system about a list of compiled object files.
*   You can't tell CGO about arbitrary directories full of C files to compile.
*   You *can* put loose C files into the package directory, but only that one
    dir -- nested dirs aren't allowed.
*   You *can* hack in linker instructions using pseudo `#cgo` directives.
    https://golang.org/cmd/cgo/#hdr-Using_cgo_with_the_go_command

We're currently using the last option to tell CGO about the static archive
built using Charmonizer/Make.

> What about code like `xs/XSBind.c`? Should it be compiled by the host, too?

Yes, IMO -- it's hard to handle otherwise because you have to know about
installation-specific include dirs and the like.

I'd should also mention that the current Python build on OSX has some
goofiness related to compilation settings.

*   Python distutils sets MACOSX_DEPLOYMENT_TARGET to an earlier version,
    deep within distutils/unixcompiler.py See
    <http://stackoverflow.com/a/33308902>.  This causes some weird warnings,
    though I eventually figured out that they seem to be harmless.
*   Python distutils compiles extensions with `-arch i386 -arch x86_64`.  This
    combination does not play well with `ar` -- you can't modify the archive,
    which breaks incremental compiling when you modify a file in `core`:
    `ar: libcfc.a is a fat file (use libtool(1) or lipo(1) and ar(1) on it)`
    `ar: libcfc.a: Inappropriate file type or format`
    The workaround is to start from scratch with a clean build.

Marvin Humphrey

Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On 24/05/2016 04:33, Marvin Humphrey wrote:
> Thus the more general strategy, which I hope will work across all hosts is:
>
> 1. Build a static archive from the `core` code.
> 2. Generate host-specific extension code wherever the host wants it to live.
> 3. Build the host extension telling the build system about various include
>    directories and telling it to link in the static archive.

There's another complication which might affect the Python and Go bindings, 
too. If you link a static library into a dynamic library with GNU binutils, 
you have to use the `--whole-archive` linker option. Otherwise, some object 
files in the static library might be ignored. So the link command should look 
something like

     gcc \
         [.o files] \
         -Wl,--whole-archive \
         libclownfish.a \
         -Wl,--no-whole-archive \
         [dynamic libs] \
         ...

The Solaris linker seems to require `-z allextract` and `-z defaultextract`. 
On OS X, we need `-all_load`. So it seems that we have to detect the linker as 
well.

>>> The tricky bit here is that these files have per-host code, as opposed to
>>> the core code which is host-agnostic.  So I actually think we should rely
>>> on the host to build them rather than the Charmonizer-generated `Makefile`.
>>
>> Makes sense. I'll change the Perl bindings to follow this approach.
>
> +1

What about code like `xs/XSBind.c`? Should it be compiled by the host, too?

Nick


Re: [lucy-dev] Separate binaries for test suites

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Sat, May 21, 2016 at 8:18 AM, Nick Wellnhofer <we...@aevum.de> wrote:
> On 20/05/2016 05:37, Marvin Humphrey wrote:
>>
>> It would also be great if we set our own flags for compiling the core code
>> and kept them the same for all hosts -- instead of compiling the core code
>> using per-host flags as we do right now.
>
> IIRC, we had this discussion already.

You're probably right but I tried to find such discussion in the dev@lucy
archives and came up empty.

> The problem is that there are certain
> C compiler flags that change the generated code in an incompatible way.

This seems similar to the HAS_BOOL issues with Perl XS extensions
(<http://markmail.org/message/5etl3pgzjkjzsxdx>).

> These compiler flags must be reused when compiling code for host language
> bindings. A good example is `-m32` to create 32-bit binaries on 64-bit
> systems. Other examples are '-mms-bitfields` (used by some Windows Perls)
> and flags concerning data alignment.
>
> So we'd have scan the host language C flags and make sure to reuse all flags
> that cause incompatible code to be generated. Since there is no conclusive
> list of all these flags, this approach is error-prone and will probably need
> occasional updates. I'm not saying it can't be done, but for now, I'd prefer
> to simply use all the compiler flags proposed by the host language. Maybe it
> would make more sense to have a blacklist of flags that are safe to remove?

My guess, which admittedly doesn't have a lot to back it up, is that although
there is an open set of potentially incompatible flags, there aren't that many
that we wouldn't find out about right away and be able to fix via
whitelisting.

It has been inconvenient to try to specify our own flags for core when there
may be conflicts with some host-specified flags.  I've run into that when
trying to specify architecture targets, disable optimizations, specify the
desired dialect of C, etc.

>> Combining them is great.  FWIW in the Python bindings I dumped the boot,
>> callbacks and binding code into a single autogenerated file, named after
>> the parcel (e.g. `python/_clownfish.c`, `python/_lucy.c`).  In the Go
>> bindings, all that stuff is in `go/PACKAGE/cfbind.go`, e.g.
>> `go/lucy/cfbind.go`.
>
>
> So these files are not under `autogen`, but somewhere else under the
> `python` or `go` directories?

Right.  Because host build systems often expect files containing extension
code to live in specific locations and they aren't amenable to custom layouts.

We actually ran into this with Module::Build, which expected only a single
source directory for C files -- necessitating a bunch of manual workaround
code to compile all our C source files.

Thus the more general strategy, which I hope will work across all hosts is:

1. Build a static archive from the `core` code.
2. Generate host-specific extension code wherever the host wants it to live.
3. Build the host extension telling the build system about various include
   directories and telling it to link in the static archive.

>> The tricky bit here is that these files have per-host code, as opposed to
>> the core code which is host-agnostic.  So I actually think we should rely
>> on the host to build them rather than the Charmonizer-generated `Makefile`.
>
> Makes sense. I'll change the Perl bindings to follow this approach.

+1

>> I would add these as separate categories:
>>
>> - static test  // ideally in `$parceldir/test` next to `$parceldir/core`
>> - static host-specific test  // maybe `$host_lang/cftest`
>>
>> (I have a slight preference for `test` for the dir where the core tests
>> file
>> live, as opposed to `tests`.)
>
>
> Another idea is to use `$parceldir/core/test` for the tests and have CFC
> ignore directories named `test` (or even any directory starting with a
> lowercase letter). This doesn't clutter up the root directory.

There's a tradeoff of shorter filepaths for test files vs. one more entry in
the package dir; I'd go for shorter filepaths.

> `$host_lang/cftest` is something we don't actually need for now.

There has actually been a small amount of test-only XS code historically, but
it's been baked into the XS binding code under
`perl/buildlib/Lucy/Build/Binding/`.

>> If we accept that there must be a 1:1 relationship between Clownfish
>> parcels and host extensions, this problem gets a lot easier. :)
>
>> Not an issue if there can only be one parcel. :) Then the only problem is
>> to build with or without including the object code for the compiled tests.
>
>
> This is a reasonable approach. Let's play it through:
>
> - We split the generated code into test and non-test files, just like
>   we would for separate binaries.
> - Then we can compile the C code and build two binaries, one with
>   the test and non-test objects, one with only non-test objects.
> - We use the test binary only to run the tests and make sure it won't
>   be installed.

Sounds good.  The devil is in the details though for host bindings, since this
isn't a configuration option they would ordinarily be expected to support.

> I think adding support for multiple parcels with separate binaries per
> parcel isn't much harder. Instead of building a binary with test and
> non-test code, we'd build a binary that contains only the test code. This
> should have other uses besides testing. OTOH, having separate binaries
> places some restrictions on the test code.

That may be, but as discussed before,
(<http://markmail.org/message/tmn4jb3yveftumu5>) Clownfish parcels were
intended to represent atomic units of installation in order to facilitate
versioning of dependencies.  Having multiple parcels installed together
complicates the mechanisms to handle resolving compatible versions and advise
what and when to upgrade.

> I have a local branch that implements separate binaries per parcel for C and
> Perl. A couple of things still have to be sorted out, mainly passing the
> correct -DCFP_* flags (this is something that's easier when having the test
> code in the same binary). I'll post it for review, then we can decide how to
> proceed. Most of the changes will be useful anyway.

OK, looking forward to it!

Marvin Humphrey

Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On 20/05/2016 05:37, Marvin Humphrey wrote:
> It would also be great if we set our own flags for compiling the core code
> and kept them the same for all hosts -- instead of compiling the core code
> using per-host flags as we do right now.

IIRC, we had this discussion already. The problem is that there are certain C 
compiler flags that change the generated code in an incompatible way. These 
compiler flags must be reused when compiling code for host language bindings. 
A good example is `-m32` to create 32-bit binaries on 64-bit systems. Other 
examples are '-mms-bitfields` (used by some Windows Perls) and flags 
concerning data alignment.

So we'd have scan the host language C flags and make sure to reuse all flags 
that cause incompatible code to be generated. Since there is no conclusive 
list of all these flags, this approach is error-prone and will probably need 
occasional updates. I'm not saying it can't be done, but for now, I'd prefer 
to simply use all the compiler flags proposed by the host language. Maybe it 
would make more sense to have a blacklist of flags that are safe to remove?

> Exactly -- `cfext` is unusual enough that it shouldn't collide with a standard
> dir for any host build system.

I realized that this isn't as pressing as I thought. But +1 for making this 
change.

> Combining them is great.  FWIW in the Python bindings I dumped the boot,
> callbacks and binding code into a single autogenerated file, named after the
> parcel (e.g. `python/_clownfish.c`, `python/_lucy.c`).  In the Go bindings,
> all that stuff is in `go/PACKAGE/cfbind.go`, e.g. `go/lucy/cfbind.go`.

So these files are not under `autogen`, but somewhere else under the `python` 
or `go` directories?

> The tricky bit here is that these files have per-host code, as opposed to the
> core code which is host-agnostic.  So I actually think we should rely on the
> host to build them rather than the Charmonizer-generated `Makefile`.

Makes sense. I'll change the Perl bindings to follow this approach.

> I would add these as separate categories:
>
> - static test  // ideally in `$parceldir/test` next to `$parceldir/core`
> - static host-specific test  // maybe `$host_lang/cftest`
>
> (I have a slight preference for `test` for the dir where the core tests file
> live, as opposed to `tests`.)

Another idea is to use `$parceldir/core/test` for the tests and have CFC 
ignore directories named `test` (or even any directory starting with a 
lowercase letter). This doesn't clutter up the root directory.

`$host_lang/cftest` is something we don't actually need for now.

> If we accept that there must be a 1:1 relationship between Clownfish parcels
> and host extensions, this problem gets a lot easier. :)

> Not an issue if there can only be one parcel. :) Then the only problem is to
> build with or without including the object code for the compiled tests.

This is a reasonable approach. Let's play it through:

- We split the generated code into test and non-test files, just like
   we would for separate binaries.
- Then we can compile the C code and build two binaries, one with
   the test and non-test objects, one with only non-test objects.
- We use the test binary only to run the tests and make sure it won't
   be installed.

I think adding support for multiple parcels with separate binaries per parcel 
isn't much harder. Instead of building a binary with test and non-test code, 
we'd build a binary that contains only the test code. This should have other 
uses besides testing. OTOH, having separate binaries places some restrictions 
on the test code.

I have a local branch that implements separate binaries per parcel for C and 
Perl. A couple of things still have to be sorted out, mainly passing the 
correct -DCFP_* flags (this is something that's easier when having the test 
code in the same binary). I'll post it for review, then we can decide how to 
proceed. Most of the changes will be useful anyway.

Nick


Re: [lucy-dev] Separate binaries for test suites

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Mon, May 16, 2016 at 3:24 AM, Nick Wellnhofer <we...@aevum.de> wrote:
> On 15/05/2016 20:38, Nick Wellnhofer wrote:
>>
>> Lucifers,
>>
>> I'd like to tackle an issue that's been nagging me for quite some time and
>> build separate binaries for our test suites. Then the test binaries can be
>> skipped when installing a Clownfish project.

This has been a TODO since the start.  It was always a kludge to link the C
test object files into the production library.

If I recall correctly, I ran into problems like:

*   which clownfish package the tests belonged to
*   which prefix the test code should use
*   assumptions about a single source dir
*   access to internals from the test files

Nothing insurmountable, but stuff that was hard to solve on the first
iteration.

It's been a while and a lot has changed -- maybe some of that is easier now.

> I like the general idea that we have a
> language-independent system to build static libraries which the host
> language's build system can link with.

Excellent.

It would also be great if we set our own flags for compiling the core code
and kept them the same for all hosts -- instead of compiling the core code
using per-host flags as we do right now.

> The main problem I see is that for now, charmonizer.exe needs a lot of
> knowledge about the locations of different source files, autogenerated or
> not, leading to code like the following that must be repeated for every
> Clownfish project:
>
> https://github.com/apache/lucy-clownfish/blob/615d033e45b6fc768e62370780e322aaa68339ac/runtime/common/charmonizer.main#L276
>
> This only gets worse if we build multiple binaries.
>
> There is an obvious improvement: Use a consistent name for the directory
> that contains host-specific C code. Currently, we have "cfext", "ext, "xs",
> and "src". I think Marvin already proposed to standardize on "cfext".

Exactly -- `cfext` is unusual enough that it shouldn't collide with a standard
dir for any host build system.

> Regarding the autogenerated C files, maybe we should create only a single
> file per parcel named after the host language. So for Perl instead of
>
>     boot.c
>     callbacks.c
>
> we have
>
>     cfish_perl.c      // Boot and callback code for the
>                       // Clownfish parcel.
>     testcfish_perl.c  // Boot and callback code for the tests.

Combining them is great.  FWIW in the Python bindings I dumped the boot,
callbacks and binding code into a single autogenerated file, named after the
parcel (e.g. `python/_clownfish.c`, `python/_lucy.c`).  In the Go bindings,
all that stuff is in `go/PACKAGE/cfbind.go`, e.g. `go/lucy/cfbind.go`.

The tricky bit here is that these files have per-host code, as opposed to the
core code which is host-agnostic.  So I actually think we should rely on the
host to build them rather than the Charmonizer-generated `Makefile`.

> For languages that don't need extra code, we can create empty files:
>
>     cfish_c.c         // Empty.
>     testcfish_c.c     // Empty.
>
> This leaves the question of how to find the source files of a parcel. Maybe
> we should stick to the rule to have a single source directory for each
> parcel? Or even enforce a directory layout involving parcel names? Then
> charmonizer.exe only needs the parcel (nick)name and the source directory to
> work out the source files for each parcel and host language:
>
>     $source_dir/*.c       // Or even core/$parcel_nick/*.c
>     $host_lang/cfext/*.c  // Must somehow handle parcels
>     autogen/source/${parcel_nick}_parcel.c
>     autogen/source/${parcel_nick}_${host_lang}.c

If we accept that there must be a 1:1 relationship between Clownfish parcels
and host extensions, this problem gets a lot easier. :)

> The remaining problem is that that the host-specific, non-generated .c files
> in "cfext" must also be assigned to the correct parcel.
>
> Speaking more generally, the .c files can be categorized as static or
> generated, and as core or host-specific, resulting in four overall
> categories:
>
> - static core
> - static host-specific
> - generated core
> - generated host-specific

I would add these as separate categories:

- static test  // ideally in `$parceldir/test` next to `$parceldir/core`
- static host-specific test  // maybe `$host_lang/cftest`

(I have a slight preference for `test` for the dir where the core tests file
live, as opposed to `tests`.)

> This again is multiplied by the number of parcels.

Not an issue if there can only be one parcel. :) Then the only problem is to
build with or without including the object code for the compiled tests.

Marvin Humphrey

Re: [lucy-dev] Separate binaries for test suites

Posted by Nick Wellnhofer <we...@aevum.de>.
On 15/05/2016 20:38, Nick Wellnhofer wrote:
> Lucifers,
>
> I'd like to tackle an issue that's been nagging me for quite some time and
> build separate binaries for our test suites. Then the test binaries can be
> skipped when installing a Clownfish project.

My original plan focused too much on the Perl side of things. Thinking more 
about it, we need a solution that works for all host languages. Maybe I should 
start by finishing my work on making the Perl build use the Makefile generated 
by charmonizer.exe. I like the general idea that we have a 
language-independent system to build static libraries which the host 
language's build system can link with.

The next step is to have the Makefile generate separate static libraries for 
the test and non-test parts of a project, or simply for each parcel. For the C 
build, we can link the static test library directly into a test executable, 
for example.

The main problem I see is that for now, charmonizer.exe needs a lot of 
knowledge about the locations of different source files, autogenerated or not, 
leading to code like the following that must be repeated for every Clownfish 
project:

 
https://github.com/apache/lucy-clownfish/blob/615d033e45b6fc768e62370780e322aaa68339ac/runtime/common/charmonizer.main#L276

This only gets worse if we build multiple binaries.

There is an obvious improvement: Use a consistent name for the directory that 
contains host-specific C code. Currently, we have "cfext", "ext, "xs", and 
"src". I think Marvin already proposed to standardize on "cfext".

Regarding the autogenerated C files, maybe we should create only a single file 
per parcel named after the host language. So for Perl instead of

     boot.c
     callbacks.c

we have

     cfish_perl.c      // Boot and callback code for the
                       // Clownfish parcel.
     testcfish_perl.c  // Boot and callback code for the tests.

For languages that don't need extra code, we can create empty files:

     cfish_c.c         // Empty.
     testcfish_c.c     // Empty.

This leaves the question of how to find the source files of a parcel. Maybe we 
should stick to the rule to have a single source directory for each parcel? Or 
even enforce a directory layout involving parcel names? Then charmonizer.exe 
only needs the parcel (nick)name and the source directory to work out the 
source files for each parcel and host language:

     $source_dir/*.c       // Or even core/$parcel_nick/*.c
     $host_lang/cfext/*.c  // Must somehow handle parcels
     autogen/source/${parcel_nick}_parcel.c
     autogen/source/${parcel_nick}_${host_lang}.c

The remaining problem is that that the host-specific, non-generated .c files 
in "cfext" must also be assigned to the correct parcel.

Speaking more generally, the .c files can be categorized as static or 
generated, and as core or host-specific, resulting in four overall categories:

- static core
- static host-specific
- generated core
- generated host-specific

This again is multiplied by the number of parcels.

Nick