Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 635 - Still Failing

[Shai Erera <se...@gmail.com>]

I committed a fix to the test - it was just wrong and tested the wrong
thing.

Shai

On Tue, Sep 30, 2014 at 5:32 PM, Shai Erera <se...@gmail.com> wrote:

> I've tried to change the test to assert better that the facet counts are
> sampled OK. E.g. the test currently computes a STDDEV, but that's wrong
> since we have only 5 categories that are sampled, and so it's not a real
> normal distribution, and every difference from the mean results in a very
> high STDDEV... So I tried Pearson's Chi-Squared test (
> http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test), which works
> better on more runs, but still fails from time to time.
>
> At that point I started thinking what exactly are we testing -- we have a
> Random sampler which samples DOCUMENTS not CATEGORIES. So e.g. if I index
> documents w/ categories in that order: A B C D A B C D A B C D and so
> forth, and I sample every 4th DOCUMENT, I could very well count only
> category D, while A, B and C might see none to very low counts, resulting
> in big STDDEV, or Chi-Square result.
>
> The sampler has no knowledge of the indexed categories, only the indexed
> documents. And you could easily write an adversary indexer which indexes
> categories such that the random sampler almost never samples some
> categories.
>
> Instead I think we should assert that the sampler sampled roughly e.g. 10%
> of the docs, irrespective of the counts of the categories. What do you
> think?
>
> On Tue, Sep 23, 2014 at 9:47 PM, Shai Erera <se...@gmail.com> wrote:
>
>> This is a test bug, but I'm not yet sure how to fix it. The test verifies
>> the sampling works OK by computing some statistics about the counted
>> facets. In particular it computes the standard deviation and ensures that
>> it's smaller than some arbitrary value (200). However, with this seed and
>> test parameters, the standard deviation is 215, and I've verified that with
>> any seed, if you fix the number of indexed documents to a high enough
>> number (>50,000), it will likely be bigger than 200.
>>
>> What I'm not sure about is how to fix the test -- increasing the number
>> from 200 to 300 will only push the limit further until another failure,
>> because of other test parameters. I can do that, and investigate again if
>> another run fails.
>>
>> But increasing that number too high misses the point I think, since if
>> our random sampling isn't really random, we'll fail to detect that.
>>
>> Basically, with some very bad luck, we could sample such that we hit the
>> maximum value of the variance, and therefore no matter the value we'll
>> compare the standard deviation to, we might run into this extremely
>> bad-luck-case at some point.
>>
>> One choice is to increase the value now, and accept that some runs may
>> fail, once in a long while ...
>>
>> Do we have other tests that do random sampling of stuff and assert the
>> sampled values?
>>
>> Shai
>>
>> On Mon, Sep 22, 2014 at 10:19 PM, Apache Jenkins Server <
>> jenkins@builds.apache.org> wrote:
>>
>>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/635/
>>>
>>> 1 tests failed.
>>> REGRESSION:
>>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling
>>>
>>> Error Message:
>>>
>>>
>>> Stack Trace:
>>> java.lang.AssertionError
>>>         at
>>> __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0)
>>>         at org.junit.Assert.fail(Assert.java:92)
>>>         at org.junit.Assert.assertTrue(Assert.java:43)
>>>         at org.junit.Assert.assertTrue(Assert.java:54)
>>>         at
>>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>>         at
>>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>>         at
>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>>         at
>>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>>         at
>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>         at
>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>         at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>>         at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>>         at
>>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>>         at
>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>>         at
>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at
>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>>>         at
>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>>         at
>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>>         at
>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>>         at
>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>>         at
>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>>
>>>
>>> Build Log:
>>> [...truncated 8590 lines...]
>>>    [junit4] Suite:
>>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector
>>>    [junit4]   2> NOTE: download the large Jenkins line-docs file by
>>> running 'ant get-jenkins-line-docs' in the lucene directory.
>>>    [junit4]   2> NOTE: reproduce with: ant test
>>> -Dtestcase=TestRandomSamplingFacetsCollector
>>> -Dtests.method=testRandomSampling -Dtests.seed=EB7A704156A4175F
>>> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
>>> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt
>>> -Dtests.locale=nl_NL -Dtests.timezone=VST -Dtests.file.encoding=ISO-8859-1
>>>    [junit4] FAILURE 21.1s J2 |
>>> TestRandomSamplingFacetsCollector.testRandomSampling <<<
>>>    [junit4]    > Throwable #1: java.lang.AssertionError
>>>    [junit4]    >        at
>>> __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0)
>>>    [junit4]    >        at
>>> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136)
>>>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>>>    [junit4]   2> NOTE: test params are: codec=Lucene410:
>>> {$payloads$=FST41, $full_path$=FST41, $facets=FSTOrd41,
>>> EvenOdd=PostingsFormat(name=Memory doPackFST= false)},
>>> docValues:{$facets=DocValuesFormat(name=Lucene410)},
>>> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {EvenOdd=DFR
>>> I(F)L1}, locale=nl_NL, timezone=VST
>>>    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation
>>> 1.7.0_65 (64-bit)/cpus=16,threads=1,free=197229168,total=297795584
>>>    [junit4]   2> NOTE: All tests run in this JVM:
>>> [TestOrdinalMappingAtomicReader, TestCachedOrdinalsReader,
>>> TestRandomSamplingFacetsCollector]
>>>    [junit4] Completed on J2 in 21.20s, 1 test, 1 failure <<< FAILURES!
>>>
>>> [...truncated 58 lines...]
>>> BUILD FAILED
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:498:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:471:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:60:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:462:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:2140:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/module-build.xml:58:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358:
>>> The following error occurred while executing this line:
>>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961:
>>> There were test failures: 23 suites, 155 tests, 1 failure
>>>
>>> Total time: 116 minutes 23 seconds
>>> Build step 'Invoke Ant' marked build as failure
>>> Archiving artifacts
>>> Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #633
>>> Archived 5 artifacts
>>> Archive block size is 32768
>>> Received 0 blocks and 228817 bytes
>>> Compression is 0.0%
>>> Took 0.11 sec
>>> Recording test results
>>> Email was triggered for: Failure
>>> Sending email for trigger: Failure
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>>
>