Re: [lucy-dev] [lucy-commits] git commit: refs/heads/cfc-tests - Start to port CFC tests to C

[Marvin Humphrey <ma...@rectangular.com>]

On Tue, Jan 29, 2013 at 11:38 AM, Nick Wellnhofer <we...@aevum.de> wrote:

> Another nice feature of TAP is that you have to split your tests into
> separate programs.  So if one of the test programs crashes, the others are
> not affected.

+1

> I thought about using TAP for testing the C bindings, but we'd also have to
> provide a TAP consumer like Perl's 'prove' utility to interpret the tests.

Sounds good.  My impression is that writing a test runner is pretty involved
in comparison to writing a TAP producer library.

> For now, I decided to go with pluggable formatters for the test output and
> provide a TAP formatter and a custom Clownfish formatter. The latter can be
> used to print a test summary from the test program directly.

It will be great if the pluggable formatter is all the scaffolding we need to
support all target platforms. :)

>> One thing I've wanted to do for a while is have our C tests return true on
>> success, false on failure, and then use that return value to determine the
>> exit value for the test script.  Right now our C-based test scripts still exit
>> success even when some tests fail.
>
> I just pushed a commit with some major reworks which also address this
> issue.

Nick++

> Another thing I'd like to have is some kind of central registry for
> tests, so we don't have to maintain code like the following in every host
> language binding:
>
>     http://s.apache.org/n3

That silly registry is only necessary because all our Lucy test classes are
inert.  If we make Lucy::Test::TestBatch subclassable[1] and have Lucy's test
classes subclass it, then we can use the following incantation:

    my $test_batch = Lucy::Test::Analysis::TestNormalizer->new(
        format => 'tap',
    );
    my $success = $test_batch->run_tests;
    exit($success ? 0 : 1);

I'd also like to propagate the work we did with Charmonizer's test harness a
while back to Clownfish/Lucy, and make the test macros operate on a hidden
global TestBatch...

    #define CFISH_TEST_OK(_expression, _message) \
        CFTest_TestBatch_Test_True(cfish_TestBatch_current, (_expression), \
                                   (_message))

... allowing us to make changes like this:

-    TEST_FALSE(batch,
-               Normalizer_Equals(normalizer[0], (Obj*)normalizer[1]),
+    TEST_FALSE(Normalizer_Equals(normalizer[0], (Obj*)normalizer[1]),
                "Equals() false with different normalization form");

Using a hidden global works fine until you need to test threads -- at which
point your tests can use full method invocations on local TestBatch objects.

Marvin Humphrey

[1] Eventually, that should be Clownfish::Test::TestBatch.