You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Niall Pemberton (JIRA)" <ji...@apache.org> on 2010/02/21 18:07:27 UTC
[jira] Created: (LANG-592) RandomUtils tests are failing frequently
RandomUtils tests are failing frequently
----------------------------------------
Key: LANG-592
URL: https://issues.apache.org/jira/browse/LANG-592
Project: Commons Lang
Issue Type: Test
Components: lang.math.*
Affects Versions: 2.x
Reporter: Niall Pemberton
Priority: Minor
Fix For: 2.5
The additionan of 40+ chi-square tests added for RandomUtils have caused the RandomUtilsTest to start failing frequently.
Phil Steitz investigated this and wrote the following (http://markmail.org/message/mo4qb3qh75nq2kwn) on the mailing list:
{code}
The random data tests are failing at a high enough frequency to be
annoying / alarming to users.
I investigated the high incidence of test failures and found nothing
wrong with what the tests are doing and nothing to indicate
systematic bias in the data being generated; but the addition of 40+
chi-square tests in the test methods added in r907159 makes the
probability of failure in a given run > 1/25. This is why there is
a high incidence of test failures.
I verified that failures appear to be evenly distributed (too many,
too few even/edd, too many, too few above/below range midpoints) and
that the chisquare statistics are being computed correctly, with the
right critical values applied.
If you do cut another RC, I would recommend one of the following:
1) Grab / copy and extend [math]'s RetryTestCase (will cut incidence
of failure in half)
2) Disable the stochastic test cases for the release
3) Reduce sensitivity of the chi-square test (change to e.g., .0005
level of significance)
4) Reduce the number of tests
My recommendation is 2) - leave in the source but comment out. The
tests are valuable as they would fail regularly and miserably if
there were systematic bias (as there used to be on odd/even); but
without reducing significantly the number of tests or the
sensitivity (or limiting to a single "successful" PRNG sequence),
there is no way to leave them all in without generating an
annoyingly high rate of random failures.
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (LANG-592) RandomUtils tests are failing
frequently
Posted by "Niall Pemberton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LANG-592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Niall Pemberton resolved LANG-592.
----------------------------------
Resolution: Fixed
Assignee: Niall Pemberton
I have moved the 40+ new chi-square tests into a new test case called RandomUtilsFreqTest and excluded that test from the default Ant/Maven test goals:
* http://svn.apache.org/viewvc?view=revision&revision=912374
I have added a new Profile to the Maven2 build which just runs the new RandomUtilsFreqTest test which can be run with the following command:
{code}
mvn -Ptest-random-freq clean test
{code}
I have added a new Ant target which just runs the new RandomUtilsFreqTest test which can be run with the following command
{code}
ant clean test-random-freq
{code}
> RandomUtils tests are failing frequently
> ----------------------------------------
>
> Key: LANG-592
> URL: https://issues.apache.org/jira/browse/LANG-592
> Project: Commons Lang
> Issue Type: Test
> Components: lang.math.*
> Affects Versions: 2.x
> Reporter: Niall Pemberton
> Assignee: Niall Pemberton
> Priority: Minor
> Fix For: 2.5
>
>
> The additionan of 40+ chi-square tests added for RandomUtils have caused the RandomUtilsTest to start failing frequently.
> Phil Steitz investigated this and wrote the following (http://markmail.org/message/mo4qb3qh75nq2kwn) on the mailing list:
> {code}
> The random data tests are failing at a high enough frequency to be
> annoying / alarming to users.
> I investigated the high incidence of test failures and found nothing
> wrong with what the tests are doing and nothing to indicate
> systematic bias in the data being generated; but the addition of 40+
> chi-square tests in the test methods added in r907159 makes the
> probability of failure in a given run > 1/25. This is why there is
> a high incidence of test failures.
> I verified that failures appear to be evenly distributed (too many,
> too few even/edd, too many, too few above/below range midpoints) and
> that the chisquare statistics are being computed correctly, with the
> right critical values applied.
> If you do cut another RC, I would recommend one of the following:
> 1) Grab / copy and extend [math]'s RetryTestCase (will cut incidence
> of failure in half)
> 2) Disable the stochastic test cases for the release
> 3) Reduce sensitivity of the chi-square test (change to e.g., .0005
> level of significance)
> 4) Reduce the number of tests
> My recommendation is 2) - leave in the source but comment out. The
> tests are valuable as they would fail regularly and miserably if
> there were systematic bias (as there used to be on odd/even); but
> without reducing significantly the number of tests or the
> sensitivity (or limiting to a single "successful" PRNG sequence),
> there is no way to leave them all in without generating an
> annoyingly high rate of random failures.
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.