You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Uwe Schindler (JIRA)" <ji...@apache.org> on 2009/02/04 15:09:59 UTC
[jira] Created: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Make tests using java.util.Random reproducible on failure
---------------------------------------------------------
Key: LUCENE-1535
URL: https://issues.apache.org/jira/browse/LUCENE-1535
Project: Lucene - Java
Issue Type: Test
Affects Versions: 2.9
Reporter: Uwe Schindler
Fix For: 2.9
This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
{code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
using the seed from the failed test printout.
*Reference:*
{quote}
: By allowing Random to randomly seed itself, we effectively test a much
: much larger space, ie every time we all run the test, it's different. We can
: potentially cast a much larger net than a fixed seed.
i guess i'm just in favor of less randomness and more iterations.
: Fixing the bug is the "easy" part; discovering a bug is present is where
: we need all the help we can get ;)
yes, but knowing a bug is there w/o having any idea what it is or how to
trigger it can be very frustrating.
it would be enough for tests to pick a random number, log it, and then use
it as the seed ... that way if you get a failure you at least know what
seed was used and you can then hardcode it temporarily to reproduce/debug
-Hoss
{quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: [jira] Created: (LUCENE-1535) Make tests using java.util.Random reproducible on failure
Posted by Uwe Schindler <uw...@thetaphi.de>.
> : Make tests using java.util.Random reproducible on failure
>
> Whoa ... i make an off the cuff comment that i forget about 10 hours
> later, and the next thing i know Uwe and Michael have made it reality.
> +1.
Thanks!
> PS: "It would be really nice if i had several million dollars, because
> that way i could spend it." (let's see what you guys can do with that one)
We see, what we can do. :)
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Created: (LUCENE-1535) Make tests using java.util.Random reproducible on failure
Posted by Michael McCandless <lu...@mikemccandless.com>.
Chris Hostetter wrote:
> PS: "It would be really nice if i had several million dollars, because
> that way i could spend it." (let's see what you guys can do with
> that one)
Alas that one is quite a bit more challenging...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Created: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by Chris Hostetter <ho...@fucit.org>.
: Make tests using java.util.Random reproducible on failure
Whoa ... i make an off the cuff comment that i forget about 10 hours
later, and the next thing i know Uwe and Michael have made it reality.
+1.
PS: "It would be really nice if i had several million dollars, because
that way i could spend it." (let's see what you guys can do with that one)
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1535:
---------------------------------------
Attachment: LUCENE-1535.patch
OK I replaced many of the "new Random(...)"'s in tests, and tweaked the impl in LuceneTestCase a bit (eg I now catch if you call newRandom twice from one test).
I think it's ready to commit. I'll wait a day or two. Maybe we'll catch ourselves a bug ;)
> Make tests using java.util.Random reproducible on failure
> ---------------------------------------------------------
>
> Key: LUCENE-1535
> URL: https://issues.apache.org/jira/browse/LUCENE-1535
> Project: Lucene - Java
> Issue Type: Test
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1535.patch, LUCENE-1535.patch
>
>
> This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
> It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
> This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
> I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
> {code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
> using the seed from the failed test printout.
> *Reference:*
> {quote}
> : By allowing Random to randomly seed itself, we effectively test a much
> : much larger space, ie every time we all run the test, it's different. We can
> : potentially cast a much larger net than a fixed seed.
> i guess i'm just in favor of less randomness and more iterations.
> : Fixing the bug is the "easy" part; discovering a bug is present is where
> : we need all the help we can get ;)
> yes, but knowing a bug is there w/o having any idea what it is or how to
> trigger it can be very frustrating.
> it would be enough for tests to pick a random number, log it, and then use
> it as the seed ... that way if you get a failure you at least know what
> seed was used and you can then hardcode it temporarily to reproduce/debug
> -Hoss
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe Schindler updated LUCENE-1535:
----------------------------------
Attachment: LUCENE-1535.patch
This is the patch.
As only TrieRangeQuery uses the new random functionality, the other tests should be scanned for usage of java.util.Random (simple grep) and be changed to use the LuceneTestCase methods to obtain a random number generator.
> Make tests using java.util.Random reproducible on failure
> ---------------------------------------------------------
>
> Key: LUCENE-1535
> URL: https://issues.apache.org/jira/browse/LUCENE-1535
> Project: Lucene - Java
> Issue Type: Test
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Fix For: 2.9
>
> Attachments: LUCENE-1535.patch
>
>
> This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
> It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
> This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
> I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
> {code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
> using the seed from the failed test printout.
> *Reference:*
> {quote}
> : By allowing Random to randomly seed itself, we effectively test a much
> : much larger space, ie every time we all run the test, it's different. We can
> : potentially cast a much larger net than a fixed seed.
> i guess i'm just in favor of less randomness and more iterations.
> : Fixing the bug is the "easy" part; discovering a bug is present is where
> : we need all the help we can get ;)
> yes, but knowing a bug is there w/o having any idea what it is or how to
> trigger it can be very frustrating.
> it would be enough for tests to pick a random number, log it, and then use
> it as the seed ... that way if you get a failure you at least know what
> seed was used and you can then hardcode it temporarily to reproduce/debug
> -Hoss
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1535.
----------------------------------------
Resolution: Fixed
Lucene Fields: [New, Patch Available] (was: [Patch Available, New])
Committed revision 741311. Thanks Uwe!
> Make tests using java.util.Random reproducible on failure
> ---------------------------------------------------------
>
> Key: LUCENE-1535
> URL: https://issues.apache.org/jira/browse/LUCENE-1535
> Project: Lucene - Java
> Issue Type: Test
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1535.patch, LUCENE-1535.patch
>
>
> This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
> It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
> This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
> I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
> {code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
> using the seed from the failed test printout.
> *Reference:*
> {quote}
> : By allowing Random to randomly seed itself, we effectively test a much
> : much larger space, ie every time we all run the test, it's different. We can
> : potentially cast a much larger net than a fixed seed.
> i guess i'm just in favor of less randomness and more iterations.
> : Fixing the bug is the "easy" part; discovering a bug is present is where
> : we need all the help we can get ;)
> yes, but knowing a bug is there w/o having any idea what it is or how to
> trigger it can be very frustrating.
> it would be enough for tests to pick a random number, log it, and then use
> it as the seed ... that way if you get a failure you at least know what
> seed was used and you can then hardcode it temporarily to reproduce/debug
> -Hoss
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1535:
------------------------------------------
Assignee: Michael McCandless
> Make tests using java.util.Random reproducible on failure
> ---------------------------------------------------------
>
> Key: LUCENE-1535
> URL: https://issues.apache.org/jira/browse/LUCENE-1535
> Project: Lucene - Java
> Issue Type: Test
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1535.patch
>
>
> This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
> It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
> This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
> I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
> {code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
> using the seed from the failed test printout.
> *Reference:*
> {quote}
> : By allowing Random to randomly seed itself, we effectively test a much
> : much larger space, ie every time we all run the test, it's different. We can
> : potentially cast a much larger net than a fixed seed.
> i guess i'm just in favor of less randomness and more iterations.
> : Fixing the bug is the "easy" part; discovering a bug is present is where
> : we need all the help we can get ;)
> yes, but knowing a bug is there w/o having any idea what it is or how to
> trigger it can be very frustrating.
> it would be enough for tests to pick a random number, log it, and then use
> it as the seed ... that way if you get a failure you at least know what
> seed was used and you can then hardcode it temporarily to reproduce/debug
> -Hoss
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by "Uwe Schindler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670421#action_12670421 ]
Uwe Schindler commented on LUCENE-1535:
---------------------------------------
Hi Mike,
all tests pass here, looks good. Thanks for the work to convert all test cases :-)
One thing (my fault): The @link reference in the javadocs to LuceneTestCase reference #getRandom(long) not #newRandom(long) -- my first version used getRandom and I renamed later.
The idea with the IllegalStateException is good and reinitializing the seed to null on each test is good (if in future jUnit changes not to create new instances for each test).
> Make tests using java.util.Random reproducible on failure
> ---------------------------------------------------------
>
> Key: LUCENE-1535
> URL: https://issues.apache.org/jira/browse/LUCENE-1535
> Project: Lucene - Java
> Issue Type: Test
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1535.patch, LUCENE-1535.patch
>
>
> This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
> It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
> This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
> I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
> {code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
> using the seed from the failed test printout.
> *Reference:*
> {quote}
> : By allowing Random to randomly seed itself, we effectively test a much
> : much larger space, ie every time we all run the test, it's different. We can
> : potentially cast a much larger net than a fixed seed.
> i guess i'm just in favor of less randomness and more iterations.
> : Fixing the bug is the "easy" part; discovering a bug is present is where
> : we need all the help we can get ;)
> yes, but knowing a bug is there w/o having any idea what it is or how to
> trigger it can be very frustrating.
> it would be enough for tests to pick a random number, log it, and then use
> it as the seed ... that way if you get a failure you at least know what
> seed was used and you can then hardcode it temporarily to reproduce/debug
> -Hoss
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1535) Make tests using java.util.Random
reproducible on failure
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670444#action_12670444 ]
Michael McCandless commented on LUCENE-1535:
--------------------------------------------
bq. The @link reference in the javadocs to LuceneTestCase reference #getRandom(long) not #newRandom(long)
No problem, I'll fix. Thanks!
bq. if in future jUnit changes not to create new instances for each test
Heh -- I had assumed it did NOT create a new instance for each test, but in fact you're right: it does.
> Make tests using java.util.Random reproducible on failure
> ---------------------------------------------------------
>
> Key: LUCENE-1535
> URL: https://issues.apache.org/jira/browse/LUCENE-1535
> Project: Lucene - Java
> Issue Type: Test
> Affects Versions: 2.9
> Reporter: Uwe Schindler
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1535.patch, LUCENE-1535.patch
>
>
> This is a patch for LuceneTestCase to support logging of the Random seed used in randomized tests. The patch also includes an example implementation in TestTrieRangeQuery.
> It overrides the protected method runTest() and inserts a try-catch around the super.runTest() call. Two new methods newRandom() and newRandom(long) are available for the test case. As each test case is run in an own TestCase object instance (so 5 test methods in a class instantiate 5 instances each method working in separate), the random seed is saved on newRandom() and when the test fails with any Throwable, a message with the seed (if not null) is printed out. If newRandom was never called no message will be printed.
> This patch has only one problem: If a single test method calls newRandom() more than once, only the last seed is saved and printed out. But each test method in a Testcase should call newRandom() exactly once for usage during the execution of this test method. And it is not thread save (no sync, no volatile), but for tests it's unimportant.
> I forgot to mention: If a test fails, the message using the seed is printed to stdout. The developer can then change the test temporarily:
> {code}LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long){code}
> using the seed from the failed test printout.
> *Reference:*
> {quote}
> : By allowing Random to randomly seed itself, we effectively test a much
> : much larger space, ie every time we all run the test, it's different. We can
> : potentially cast a much larger net than a fixed seed.
> i guess i'm just in favor of less randomness and more iterations.
> : Fixing the bug is the "easy" part; discovering a bug is present is where
> : we need all the help we can get ;)
> yes, but knowing a bug is there w/o having any idea what it is or how to
> trigger it can be very frustrating.
> it would be enough for tests to pick a random number, log it, and then use
> it as the seed ... that way if you get a failure you at least know what
> seed was used and you can then hardcode it temporarily to reproduce/debug
> -Hoss
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org