You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Robert Muir (JIRA)" <ji...@apache.org> on 2010/05/06 19:44:49 UTC
[jira] Commented: (LUCENE-2449) Improve random testing
[ https://issues.apache.org/jira/browse/LUCENE-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864855#action_12864855 ]
Robert Muir commented on LUCENE-2449:
-------------------------------------
I ran the patch with -Drandom.multiplier=23
The tests spun and spun (i finally killed it), but i produced a few failures. Hopefully they are all just bugs I introduced into the tests!
After 2 minutes, TestIW failed here:
{noformat}
[junit] Testsuite: org.apache.lucene.index.TestIndexWriter
[junit] Testcase: testTermUTF16SortOrder(org.apache.lucene.index.TestIndexWriter): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.MultiTermsEnum.seek(MultiTermsEnum.java:143)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.TestIndexWriter.checkTermsOrder(TestIndexWriter.java:4717)
[junit] at org.apache.lucene.index.TestIndexWriter.testTermUTF16SortOrder(TestIndexWriter.java:4789)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:276)
[junit]
[junit]
[junit] Tests run: 112, Failures: 1, Errors: 0, Time elapsed: 143.461 sec
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testTermUTF16SortOrder' was: 1356727775332805472
{noformat}
All NRQ tests failed, i must have screwed up the tests really bad
{noformat}
[junit] Testsuite: org.apache.lucene.search.TestMultiValuedNumericRangeQuery
[junit] Testcase: testMultiValuedNRQ(org.apache.lucene.search.TestMultiValuedNumericRangeQuery): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:98)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:108)
[junit] at org.apache.lucene.search.TestMultiValuedNumericRangeQuery.testMultiValuedNRQ(TestMultiValuedNumericRangeQuery.java:72)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:276)
[junit]
[junit]
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 74.553 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testMultiValuedNRQ' was: 6379316497871210234
[junit] ------------- ---------------- ---------------
[junit] Testsuite: org.apache.lucene.search.TestNumericRangeQuery64
[junit] Testcase: testRandomTrieAndClassicRangeQuery_6bit(org.apache.lucene.search.TestNumericRangeQuery64):FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:98)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:108)
[junit] at org.apache.lucene.search.TestNumericRangeQuery64.testRandomTrieAndClassicRangeQuery(TestNumericRangeQuery64.java:364)
[junit] at org.apache.lucene.search.TestNumericRangeQuery64.testRandomTrieAndClassicRangeQuery_6bit(TestNumericRangeQuery64.java:410)
[junit]
[junit]
[junit] Testcase: testSorting_6bit(org.apache.lucene.search.TestNumericRangeQuery64): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:49)
[junit] at org.apache.lucene.search.TestNumericRangeQuery64.testSorting(TestNumericRangeQuery64.java:525)
[junit] at org.apache.lucene.search.TestNumericRangeQuery64.testSorting_6bit(TestNumericRangeQuery64.java:545)
[junit]
[junit]
[junit] Tests run: 34, Failures: 2, Errors: 0, Time elapsed: 544.22 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testRandomTrieAndClassicRangeQuery_6bit' was: 3692414376588312592
[junit] NOTE: random seed of testcase 'testSorting_6bit' was: -5115416782250700412
[junit] ------------- ---------------- ---------------
[junit] Testsuite: org.apache.lucene.search.TestNumericRangeQuery32
[junit] Testcase: testRandomTrieAndClassicRangeQuery_4bit(org.apache.lucene.search.TestNumericRangeQuery32):FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:98)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:108)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testRandomTrieAndClassicRangeQuery(TestNumericRangeQuery32.java:346)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testRandomTrieAndClassicRangeQuery_4bit(TestNumericRangeQuery32.java:392)
[junit]
[junit]
[junit] Testcase: testRangeSplit_8bit(org.apache.lucene.search.TestNumericRangeQuery32): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:98)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:108)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testRangeSplit(TestNumericRangeQuery32.java:421)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testRangeSplit_8bit(TestNumericRangeQuery32.java:436)
[junit]
[junit]
[junit] Testcase: testRangeSplit_2bit(org.apache.lucene.search.TestNumericRangeQuery32): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:98)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:108)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testRangeSplit(TestNumericRangeQuery32.java:417)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testRangeSplit_2bit(TestNumericRangeQuery32.java:446)
[junit]
[junit]
[junit] Testcase: testSorting_8bit(org.apache.lucene.search.TestNumericRangeQuery32): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:49)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testSorting(TestNumericRangeQuery32.java:492)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testSorting_8bit(TestNumericRangeQuery32.java:507)
[junit]
[junit]
[junit] Testcase: testSorting_4bit(org.apache.lucene.search.TestNumericRangeQuery32): FAILED
[junit] null
[junit] junit.framework.AssertionFailedError
[junit] at org.apache.lucene.index.codecs.standard.StandardTermsDictReader$FieldReader$SegmentTermsEnum.seek(StandardTermsDictReader.java:377)
[junit] at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
[junit] at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
[junit] at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:935)
[junit] at org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:724)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite$CutOffTermCollector.collect(MultiTermQuery.java:609)
[junit] at org.apache.lucene.search.MultiTermQuery$BooleanQueryRewrite.collectTerms(MultiTermQuery.java:223)
[junit] at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:568)
[junit] at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:755)
[junit] at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:270)
[junit] at org.apache.lucene.search.Query.weight(Query.java:100)
[junit] at org.apache.lucene.search.Searcher.createWeight(Searcher.java:147)
[junit] at org.apache.lucene.search.Searcher.search(Searcher.java:49)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testSorting(TestNumericRangeQuery32.java:492)
[junit] at org.apache.lucene.search.TestNumericRangeQuery32.testSorting_4bit(TestNumericRangeQuery32.java:512)
[junit]
[junit]
[junit] Tests run: 27, Failures: 5, Errors: 0, Time elapsed: 332.621 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testRandomTrieAndClassicRangeQuery_4bit' was: -2669854942242274524
[junit] NOTE: random seed of testcase 'testRangeSplit_8bit' was: 5368342330535968370
[junit] NOTE: random seed of testcase 'testRangeSplit_2bit' was: 4707437135047159199
[junit] NOTE: random seed of testcase 'testSorting_8bit' was: 4999254898192507629
[junit] NOTE: random seed of testcase 'testSorting_4bit' was: 3362223988444615041
[junit] ------------- ---------------- ---------------
{noformat}
TestRegexpRandom2 failed
{noformat}
[junit] Testsuite: org.apache.lucene.search.TestRegexpRandom2
[junit] Testcase: testRegexps(org.apache.lucene.search.TestRegexpRandom2): FAILED
[junit] for re:. expected:<2808> but was:<2801>
[junit] junit.framework.AssertionFailedError: for re:. expected:<2808> but was:<2801>
[junit] at org.apache.lucene.search.TestRegexpRandom2.assertSame(TestRegexpRandom2.java:138)
[junit] at org.apache.lucene.search.TestRegexpRandom2.testRegexps(TestRegexpRandom2.java:117)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:276)
[junit]
[junit]
[junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 125.146 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testRegexps' was: 8221008363438780882
[junit] ------------- ---------------- ---------------
{noformat}
The automaton convertor failed, probably some illegal surrogate case
{noformat}
[junit] Testsuite: org.apache.lucene.util.automaton.TestUTF32ToUTF8
[junit] Testcase: testRandomRegexes(org.apache.lucene.util.automaton.TestUTF32ToUTF8): Caused an ERROR
[junit] 2361
[junit] java.lang.ArrayIndexOutOfBoundsException: 2361
[junit] at java.lang.Character.toSurrogates(Character.java:2596)
[junit] at java.lang.String.<init>(String.java:275)
[junit] at org.apache.lucene.util.automaton.TestUTF32ToUTF8.assertAutomaton(TestUTF32ToUTF8.java:186)
[junit] at org.apache.lucene.util.automaton.TestUTF32ToUTF8.testRandomRegexes(TestUTF32ToUTF8.java:170)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:276)
[junit]
[junit]
[junit] Tests run: 5, Failures: 0, Errors: 1, Time elapsed: 14.099 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testRandomRegexes' was: -6824078967033722523
[junit] ------------- ---------------- ---------------
{noformat}
for TestDeterminizeLexicon, it just ran out of memory (we should probably put a Math.min on lexicon size, no matter the multiplier)
{noformat}
[junit] Testsuite: org.apache.lucene.util.automaton.TestDeterminizeLexicon
[junit] Testcase: testLexicon(org.apache.lucene.util.automaton.TestDeterminizeLexicon): Caused an ERROR
[junit] Java heap space
[junit] java.lang.OutOfMemoryError: Java heap space
[junit] at org.apache.lucene.util.automaton.UTF32ToUTF8.newUTF8State(UTF32ToUTF8.java:302)
[junit] at org.apache.lucene.util.automaton.UTF32ToUTF8.convert(UTF32ToUTF8.java:285)
[junit] at org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:23)
[junit] at org.apache.lucene.util.automaton.TestDeterminizeLexicon.assertLexicon(TestDeterminizeLexicon.java:63)
[junit] at org.apache.lucene.util.automaton.TestDeterminizeLexicon.testLexicon(TestDeterminizeLexicon.java:51)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:276)
[junit]
[junit]
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 33.064 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] NOTE: random seed of testcase 'testLexicon' was: 2016020851907753137
[junit] ------------- ---------------- ---------------
{noformat}
> Improve random testing
> ----------------------
>
> Key: LUCENE-2449
> URL: https://issues.apache.org/jira/browse/LUCENE-2449
> Project: Lucene - Java
> Issue Type: Test
> Components: Build
> Reporter: Robert Muir
> Priority: Minor
> Attachments: LUCENE-2449.patch, LUCENE-2449.patch
>
>
> We have quite a few random tests, but there's no way to "crank" them.
> The idea here is to add a multiplier which can be increased by a sysprop. For example, we could set this to something higher than 1 for hudson.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org