You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2020/06/24 18:13:14 UTC

[GitHub] [lucenenet] NightOwl888 opened a new issue #288: Fix Random Seed Functionality in TestFramework

NightOwl888 opened a new issue #288:
URL: https://github.com/apache/lucenenet/issues/288


   The random seed that is set to `StringHelper.GOOD_FAST_HASH_SEED` through the `tests.seed` system property in Java currently is the same as the seed that NUnit uses in `NUnit.Framework.TestContext.CurrentContext.Random`. If we continue using NUnit's `Random` property, more research is needed to figure out how to set that seed the way the NUnit designers intended so it can be documented for end users.
   
   During testing, I noticed that NUnit does set the seed to a consistent pseudo-random value, so the same test can be run over and over with exactly the same random setup. However, the seed is re-generated when the code in the project under test is changed. I noted that NUnit writes the seed to a file named `nunit_random_seed.tmp` in the `bin` folder of the test project when run in Visual Studio.
   
   ### A Typical Testing Scenario in Lucene
   
   1. Test fails
   2. Test failure generates an error message that includes the random seed as a hexadecimal string (that represents a long)
   3. Developer sets the `tests.seed` system property to the same hexadecimal string that caused the test to fail
   4. Debugging can take place because the pseudo-randomly generated conditions are exactly the same as they were when the test failed
   5. A fix is devised
   
    ### Testing in .NET
   
   All we have been able to do is get the same test to fail multiple times, but that breaks as soon as any code is changed. We could probably revert back if we save a copy of the `nunit_random_seed.tmp`, but this is very complicated to do in addition to debugging, and not very intuitive.
   
   We are missing:
   
   1. The test framework reporting the seed that it was using when the test failed
   2. The ability to set the random seed
   
   I suspect that although setting the seed cannot be done publicly, we can hack it by using .NET Reflection to set an internal field. There was some [discussion about setting it through an attribute](https://github.com/nunit/nunit/issues/1461), but it doesn't sound like that has been implemented.
   
   I haven't looked into what it will take to extend NUnit to attach the random seed to the test message that is reported. One option (but not a very good one) would be to change the `Lucene.Net.TestFramework.Assert` class to add the random seed to all of test messages. The main thing we need is to read the seed value. The only place it seems to be exposed is in the `bin/nunit_random_seed.tmp` file, but it would be preferable to read it through a property within the test session.
   
   ### Testing Lucene.NET Itself
   
   For end users, the above would solve all of the issues. However, for Lucene.NET we often need to generate the same conditions as Java to determine where the execution paths diverge. This is a problem because the NUnit Random class (probably) isn't the same implementation as the Java Random class. Also, in Java the random seed is a `long`, but in .NET, it is an `int`.
   
   We have ported the Java Random class in `J2N`, named [Randomizer](https://github.com/NightOwl888/J2N/blob/master/src/J2N/Randomizer.cs). Ideally, we would use this implementation during testing. However, NUnit doesn't seem to have a way to inject a custom implementation of `Random` nor does it have a way to read the seed it uses to seed a new instance of `Randomizer` for the test framework or a way to override that setting manually during debugging.
   
   ### Ideal Solution
   
   1. By default, the test framework uses a seed generated by the same hook that NUnit generates their seed. Alternatively, we could use `System.DateTime.UtcNow.Ticks`.
   2. The seed generated would be a `long`.
   3. The generated seed would be used to seed the `J2N.Randomizer` class, which the getter of the `LuceneTestFramework.Random` property would provide and cache.
   4. The test framework would ensure the seed is always output along with failure test messages, as a hexadecimal string.
   5. The "System Properties" feature allows `tests:seed` to be set to a hexadecimal string, which if set would override the auto-generated random seed used in step 3.
   
   Note that the ideal solution doesn't necessarily involve NUnit in the equation. With Java's Random class ported and "System Properties" solved in a way that doesn't involve hacking the system's environment variables when debugging, we are much closer to fixing this to make it compatible with Java Lucene so we can test apples to apples.
   
   The main thing we are missing is writing the random seed out into the test message with hexadecimal formatting (the same string that is used in Lucene). Other than that, setting up the system property to override the automatically generated seed should be fairly straightforward.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [lucenenet] NightOwl888 closed issue #288: Fix Random Seed Functionality in TestFramework

Posted by GitBox <gi...@apache.org>.
NightOwl888 closed issue #288:
URL: https://github.com/apache/lucenenet/issues/288


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@lucenenet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [lucenenet] NightOwl888 closed issue #288: Fix Random Seed Functionality in TestFramework

Posted by GitBox <gi...@apache.org>.
NightOwl888 closed issue #288:
URL: https://github.com/apache/lucenenet/issues/288


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [lucenenet] NightOwl888 commented on issue #288: Fix Random Seed Functionality in TestFramework

Posted by GitBox <gi...@apache.org>.
NightOwl888 commented on issue #288:
URL: https://github.com/apache/lucenenet/issues/288#issuecomment-783687320


   Great. I have assigned it to you.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [lucenenet] rclabo commented on issue #288: Fix Random Seed Functionality in TestFramework

Posted by GitBox <gi...@apache.org>.
rclabo commented on issue #288:
URL: https://github.com/apache/lucenenet/issues/288#issuecomment-783684654


   I'd like to tackle this one.  Should be a good learning exercise.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org