Question: unittest use of assert (peter??9

[jan i <ja...@apache.org>]

Hi.

This is probably a question for Peter.

I am currently writing unit tests using the new harness from Peter (which
is really nice). I did extent -plain a little bit.

However all test functions are "void foo()", so if a test fails the
functions does an assert. This means the test run will stop at the first
error.

Would it not be better to have "bool fool()", so a test run would run all
tests, meaning a tester might see multiple errors in one run.

Thoughts ?

rgds
jan I.

Ps. I dont have a strong opinion here, I am just used to return codes.

Re: Question: unittest use of assert (peter??9

[jan i <ja...@apache.org>]

On Friday, December 26, 2014, Peter Kelly <pm...@apache.org> wrote:

> > On 26 Dec 2014, at 4:32 am, jan i <jani@apache.org <javascript:;>>
> wrote:
> >
> > Hi.
> >
> > This is probably a question for Peter.
> >
> > I am currently writing unit tests using the new harness from Peter (which
> > is really nice). I did extent -plain a little bit.
> >
> > However all test functions are "void foo()", so if a test fails the
> > functions does an assert. This means the test run will stop at the first
> > error.
> >
> > Would it not be better to have "bool fool()", so a test run would run all
> > tests, meaning a tester might see multiple errors in one run.
>
> There’s two different assertion functions:
>
> assert - provided by the C library. This aborts the program on execution
> utassert - provided by DocFormats itself (defined in the unittest
> directory). This just causes a test to fail, without aborting execution of
> the program

I missed the difference thanks for the explanation.

>
> Assert is used in a number of places throughout the code in the main part
> of the library, for the usual sanity checks. I would suggest leaving these
> as-is, because an assertion failure in these places typically means that if
> it’s ignored then the program will crash or otherwise corrupt memory or
> something similarly bad anyway, just slightly later (the assertion catches
> the problem before it happens).
>
> utassert is intended for use in test cases only, where you want to ensure
> a particular condition is true. After calling it, execution of the test
> function will continue, but the result of the test would be failure.
>
> If we want to have a test harness that allows tests where the former
> assertion fails (causing the process to abort), we can do so by forking
> between each test (though this will not work on windows however, which
> lacks fork). This would also be helpful for problems which lead to NULL
> pointer de-references or memory corruption; each test would run in complete
> isolation from the other. This would be relatively easy to do and I can add
> this if you want (or you can do so yourself if you wish).
>
> I don’t personally feel a need for such a case, as far as my own
> development work goes. If I’m working away and introduce a bug that’s
> serious enough to cause a crash, I’ll generally fix it immediately before
> continuing. However if anyone else wants to have the isolation I’ve
> described above so that it can try later tests even if an earlier causes a
> crash (or assert() abortion - note this is distinct from utassert()) then
> this could be added to the test harness.

I agree with you an abort like NULL pointer should lead to a stop, and with
utassert test cases can fail for logical reasons without stopping other
test cases.

I will leave the test harness as it is, and I had no intention of changing
the assert in the production code
rgds
jan i

>
> —
> Dr Peter M. Kelly
> pmkelly@apache.org <javascript:;>
>
> PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
> (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
>
>

-- 
Sent from My iPad, sorry for any misspellings.

Re: Question: unittest use of assert (peter??9

[Peter Kelly <pm...@apache.org>]

> On 26 Dec 2014, at 4:32 am, jan i <ja...@apache.org> wrote:
> 
> Hi.
> 
> This is probably a question for Peter.
> 
> I am currently writing unit tests using the new harness from Peter (which
> is really nice). I did extent -plain a little bit.
> 
> However all test functions are "void foo()", so if a test fails the
> functions does an assert. This means the test run will stop at the first
> error.
> 
> Would it not be better to have "bool fool()", so a test run would run all
> tests, meaning a tester might see multiple errors in one run.

There’s two different assertion functions:

assert - provided by the C library. This aborts the program on execution
utassert - provided by DocFormats itself (defined in the unittest directory). This just causes a test to fail, without aborting execution of the program

Assert is used in a number of places throughout the code in the main part of the library, for the usual sanity checks. I would suggest leaving these as-is, because an assertion failure in these places typically means that if it’s ignored then the program will crash or otherwise corrupt memory or something similarly bad anyway, just slightly later (the assertion catches the problem before it happens).

utassert is intended for use in test cases only, where you want to ensure a particular condition is true. After calling it, execution of the test function will continue, but the result of the test would be failure.

If we want to have a test harness that allows tests where the former assertion fails (causing the process to abort), we can do so by forking between each test (though this will not work on windows however, which lacks fork). This would also be helpful for problems which lead to NULL pointer de-references or memory corruption; each test would run in complete isolation from the other. This would be relatively easy to do and I can add this if you want (or you can do so yourself if you wish).

I don’t personally feel a need for such a case, as far as my own development work goes. If I’m working away and introduce a bug that’s serious enough to cause a crash, I’ll generally fix it immediately before continuing. However if anyone else wants to have the isolation I’ve described above so that it can try later tests even if an earlier causes a crash (or assert() abortion - note this is distinct from utassert()) then this could be added to the test harness.

—
Dr Peter M. Kelly
pmkelly@apache.org

PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
(fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)