Re: [lucy-dev] [lucy-commits] git commit: refs/heads/cfc-tests - Start to port CFC tests to C

[Nick Wellnhofer <we...@aevum.de>]

On Jan 29, 2013, at 03:39 , Marvin Humphrey <ma...@rectangular.com> wrote:

> Dreaming big dreams... I'd love to have a test harness that supported test
> classes like xUnit, but ran tests that don't throw exceptions, a la Perl tests
> instead of xUnit assertions.  Most of the time, a failing test does not
> invalidate subsequent tests -- and having additional information from tests
> which either succeed or fail can be a big help when trying to debug a CPAN
> Testers report where you don't have access to the machine on which the tests
> ran.

Another nice feature of TAP is that you have to split your tests into separate programs. So if one of the test programs crashes, the others are not affected.

I thought about using TAP for testing the C bindings, but we'd also have to provide a TAP consumer like Perl's 'prove' utility to interpret the tests. For now, I decided to go with pluggable formatters for the test output and provide a TAP formatter and a custom Clownfish formatter. The latter can be used to print a test summary from the test program directly.

>> +$test->run_batch('Clownfish::CFC::Util');
>> +
> 
> One thing I've wanted to do for a while is have our C tests return true on
> success, false on failure, and then use that return value to determine the
> exit value for the test script.  Right now our C-based test scripts still exit
> success even when some tests fail.

I just pushed a commit with some major reworks which also address this issue. Another thing I'd like to have is some kind of central registry for tests, so we don't have to maintain code like the following in every host language binding:

    http://s.apache.org/n3

For the CFC tests, I simply put all the needed information in a global array.

Nick

Re: [lucy-dev] [lucy-commits] git commit: refs/heads/cfc-tests - Start to port CFC tests to C

[Marvin Humphrey <ma...@rectangular.com>]

On Tue, Jan 29, 2013 at 11:38 AM, Nick Wellnhofer <we...@aevum.de> wrote:

> Another nice feature of TAP is that you have to split your tests into
> separate programs.  So if one of the test programs crashes, the others are
> not affected.

+1

> I thought about using TAP for testing the C bindings, but we'd also have to
> provide a TAP consumer like Perl's 'prove' utility to interpret the tests.

Sounds good.  My impression is that writing a test runner is pretty involved
in comparison to writing a TAP producer library.

> For now, I decided to go with pluggable formatters for the test output and
> provide a TAP formatter and a custom Clownfish formatter. The latter can be
> used to print a test summary from the test program directly.

It will be great if the pluggable formatter is all the scaffolding we need to
support all target platforms. :)

>> One thing I've wanted to do for a while is have our C tests return true on
>> success, false on failure, and then use that return value to determine the
>> exit value for the test script.  Right now our C-based test scripts still exit
>> success even when some tests fail.
>
> I just pushed a commit with some major reworks which also address this
> issue.

Nick++

> Another thing I'd like to have is some kind of central registry for
> tests, so we don't have to maintain code like the following in every host
> language binding:
>
>     http://s.apache.org/n3

That silly registry is only necessary because all our Lucy test classes are
inert.  If we make Lucy::Test::TestBatch subclassable[1] and have Lucy's test
classes subclass it, then we can use the following incantation:

    my $test_batch = Lucy::Test::Analysis::TestNormalizer->new(
        format => 'tap',
    );
    my $success = $test_batch->run_tests;
    exit($success ? 0 : 1);

I'd also like to propagate the work we did with Charmonizer's test harness a
while back to Clownfish/Lucy, and make the test macros operate on a hidden
global TestBatch...

    #define CFISH_TEST_OK(_expression, _message) \
        CFTest_TestBatch_Test_True(cfish_TestBatch_current, (_expression), \
                                   (_message))

... allowing us to make changes like this:

-    TEST_FALSE(batch,
-               Normalizer_Equals(normalizer[0], (Obj*)normalizer[1]),
+    TEST_FALSE(Normalizer_Equals(normalizer[0], (Obj*)normalizer[1]),
                "Equals() false with different normalization form");

Using a hidden global works fine until you need to test threads -- at which
point your tests can use full method invocations on local TestBatch objects.

Marvin Humphrey

[1] Eventually, that should be Clownfish::Test::TestBatch.