[jira] [Resolved] (LUCENE-3492) Extract a generic framework for running randomized tests.

["Dawid Weiss (Resolved) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dawid Weiss resolved LUCENE-3492.
---------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 4.0)

I consider this issue done. I've pulled all the super-cool features from Lucene/Solr and put them into a separate, stand-alone and reusable project called randomizedtesting. We have switched our infrastructure at Carrot2 to use it and I'm very happy with it.

http://labs.carrotsearch.com/randomizedtesting.html

I will file another issue concerned with progressively moving from LuceneTestRunner/Case and tests infrastructure to RandomizedRunner (and siblings).
                
> Extract a generic framework for running randomized tests.
> ---------------------------------------------------------
>
>                 Key: LUCENE-3492
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3492
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: general/test
>            Reporter: Dawid Weiss
>            Assignee: Dawid Weiss
>            Priority: Minor
>         Attachments: Screen Shot 2011-10-06 at 12.58.02 PM.png
>
>
> {color:red}The work on this issue is temporarily at github{color} (lots of experiments and tweaking):
> https://github.com/carrotsearch/randomizedtesting
> Or directly: git clone git://github.com/carrotsearch/randomizedtesting.git
> {color}
> ----
> RandomizedRunner is a JUnit runner, so it is capable of running @Test-annotated test cases. It
> respects regular lifecycle hooks such as @Before, @After, @BeforeClass or @AfterClass, but it
> also adds the following:
> Randomized, but repeatable execution and infrastructure for dealing with randomness:
> - uses pseudo-randomness (so that a given run can be repeated if given the same starting seed)
>   for many things called "random" below,
> - randomly shuffles test methods to ensure they don't depend on each other,
> - randomly shuffles hooks (within a given class) to ensure they don't depend on each other,
> - base class RandomizedTest provides a number of methods for generating random numbers, strings
>   and picking random objects from collections (again, this is fully repeatable given the
>   initial seed if there are no race conditions),
> - the runner provides infrastructure to augment stack traces with information about the initial
>   seeds used for running the test, so that it can be repeated (or it can be determined that
>   the test is not repeatable -- this indicates a problem with the test case itself).
> Thread control:
> - any threads created as part of a test case are assigned the same initial random seed 
>   (repeatability),
> - tracks and attempts to terminate any Threads that are created and not terminated inside 
>   a test case (not cleaning up causes a test failure),
> - tracks and attempts to terminate test cases that run for too long (default timeout: 60 seconds,
>   adjustable using global property or annotations),
> Improved validation and lifecycle support:
> - RandomizedRunner uses relaxed contracts of hook methods' accessibility (hook methods _can_ be
>   private). This helps in avoiding problems with method shadowing (static hooks) or overrides
>   that require tedious super.() chaining). Private hooks are always executed and don't affect
>   subclasses in any way, period.
> - @Listeners annotation on a test class allows it to hook into the execution progress and listen
>   to events.
> - @Validators annotation allows a test class to provide custom validation strategies 
>   (project-specific). For example a base class can request specific test case naming strategy
>   (or reject JUnit3-like methods, for instance).
> - RandomizedRunner does not "chain" or "suppress" exceptions happening during execution of 
>   a test case (including hooks). All exceptions are reported as soon as they happened and multiple
>   failure reports can occur. Most environments we know of then display these failures sequentially
>   allowing a clearer understanding of what actually happened first.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org