[GitHub] [lucenenet] NightOwl888 opened a new pull request #547: Lucene.Net.TestFramework: Fixed random seed functionality so it is repeatable (Fixes #288)

[GitBox <gi...@apache.org>]

NightOwl888 opened a new pull request #547:
URL: https://github.com/apache/lucenenet/pull/547


   > NOTE: This PR should be squashed.
   
   Fixes #288.
   
   These changes are still experimental, have had minimal manual testing, and don't have any automated tests. But, the need to repeat a test failure outweighs the need to make these features stable before merging them, IMHO.
   
   ## Features
   
   - Random Seed Reporting - Upon a test failure, the random seed, random culture, and instructions on how to apply it are appended to the test message.
   - Test Fixture Wrapping - We add an extra outer nested test fixture to capture all of the `OneTimeSetUp` and `OneTimeTearDown` calls before the first test and after the last test in the fixture.
   - `LuceneTestCase.TestFixtureAttribute` - A copy of NUnit's attribute, modified with some extra hooks to setup our test framework.
   - `LuceneRandomSeedInitializer` - Determines whether to use a random seed or one from the current context to initialize the seeds for each test fixture/test. Also registers the `RandomizedContext` for each test fixture/test.
   - `RandomizedContext` - A static class that is registered as an NUnit test property, which allows us to use strongly-typed properties for `RandomSeed` (the initial seed), `TestSeed`, `CurrentTestAssembly`, and of course the `RandomGenerator` for the context.
   - `LuceneTestCase.SetUpFixture` - An internal class that captures all `OneTimeSetUp` and `OneTimeTearDown` calls from our "wrapper" test fixture and uses reference counting to ensure we always call `LuceneTestFrameworkInitializer` (or a subclass of it) exactly 1 time before the first test is run and after the last test is run.
   - `NUnitTestFixtureBuilder` - A copy of the class with the same name from NUnit, with the extra bits to inject the random seeds and take the filters out of the equation so the random generator gets the exact same number of calls regardless of which filters are applied. It also manages the assembly-level `RandomizedContext` instance, which is applied as a property to every "wrapper" test fixture.
   - `RandomSeedAttribute` - An assembly-level attribute that can be used to apply a random seed that was reported as a failure to repeat the results. It is also possible to set the random seed in a .runsettings file in the solution directory or a `lucene.testsettings.json` file in any directory in the path of the test assembly.
   - `UseTempLineDocsFileAttribute` - An assembly-level or class-level attribute that specifies to unzip the `LineFileDocs` to a temp directory one time per assembly, so it doesn't have to be repeated in each test. This feature only works if the file has a `.gz` extension. The path to the unzipped file is available from the `LuceneTestCase.TestLineDocsFile` property. To specify an external `LineFileDocs` file, use the `tests:linedocsfile` system property (which works from .runsettings or from lucene.testsettings.json).
   - `NUnit.Framework.SetCultureAttribute` - This attribute can be used to override the randomly generated culture. Or it can be set in .runsettings/lucene.testsettings.json using the key `tests:culture` and the value of the culture (such as `fr-CA`).
   
   ## Breaking Changes
   
   - An `Initialize()` virtual method was added to `LuceneTestFrameworkInitializer`, which is where the properties `CodecFactory`, `DocValuesFormatFactory`, `PostingsFormatFactory`, and `ConfigurationFactory` must be set. An `InvalidOperationException` will be thrown if they are set in the constructor, `TestFrameworkSetUp()` or `TestFrameworkTearDown()`.
   - `StringHelper.GOOD_FAST_HASH_SEED` has been turned into a static property and renamed to `StringHelper.GoodFastHashSeed` (the seed is also a setting that is set to a fixed value during repeatable testing).
   - `LuceneTestCase.GetTestClass()` was deprecated and changed into a new property `TestType`.
   - `AbstractBeforeAfterRule.Before()` and `AbstractBeforeAfterRule.After()` methods have had their `LuceneTestCase` parameter removed.
   
   This PR also includes some plumbing to generate the source code to build a `SynonymMap`, so it can be quickly reconstituted for a non-random test (for internal debugging purposes).
   
   The `[AwaitsFix]` attribute has been applied to the remaining test failures, and the `azure-pipelines.yml` is now set to fail the build if we get more than 0 test failures (previously this was set to 2).
   
   ## References
   
   - https://get-testy.com/tag/nunit/
   - https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [lucenenet] NightOwl888 merged pull request #547: Lucene.Net.TestFramework: Fixed random seed functionality so it is repeatable (Fixes #288)

[GitBox <gi...@apache.org>]

NightOwl888 merged pull request #547:
URL: https://github.com/apache/lucenenet/pull/547


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org