[RNG] Intermittent failure of unit tests (Was: [VOTE][RC5] ...)

[Gilles <gi...@harfang.homelinux.org>]

On Thu, 1 Dec 2016 15:59:24 -0800, Rob Tompkins wrote:
>> On Dec 1, 2016, at 3:32 PM, Gilles <gi...@harfang.homelinux.org> 
>> wrote:
>>
>> On Thu, 1 Dec 2016 15:29:17 -0800, Rob Tompkins wrote:
>>> Vote: -1 mainly for license,
>>
>> Which files?
>>
>>> but there\u2019s the following as well.
>>>
>>> I just ran \u201cmvn clean test" with java 1.7.0_60 on OS X 10.12.1 and
>>> got the following test failure (note that it seems to be
>>> intermittent):
>>
>> Cf. https://issues.apache.org/jira/browse/RNG-30 
>> <https://issues.apache.org/jira/browse/RNG-30>
>
> I don\u2019t know what to do here. On one hand it really feels like our
> builds should be deterministic, but on the other hand you need to 
> need
> to exercise the code fully in your unit tests. I\u2019m really at odds 
> with
> myself, but I think I lean towards determinism.

As with other issues discussed on this list, this one is not going
to be solved (reasonably, I mean) by tallying "preferences".[1]

It is possible that the unit test's assumption is flawed.[2]
Another possiblity is that the strategy for declaring pass/fail is
too crude; see for example, in the user guide, the output of the
"Dieharder" test suite.[3]


Regards,
Gilles

[1] It is easy to select a seed that will make the test deterministic 
and
     passing.  But it suppresses behavioral diversity, which in turn 
could
     reveal a potential bug.  This case seems to happen in
       https://issues.apache.org/jira/browse/MATH-1361
[2] Hence the request for review in "RNG-30".
[3] AFAIUC, when a test produces a result that is "close" to the 
pass/fail
     boundary, it is rerun, until a definite conclusion can be reached.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org