[jira] [Comment Edited] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed

["Joel Bernstein (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967569#comment-16967569 ] 

Joel Bernstein edited comment on SOLR-13864 at 11/5/19 2:30 PM:
----------------------------------------------------------------

I'll fix this. I went overboard on these tests. No need to test that the random behavior is as expected. Just need to test the function works and it brings back results that look right. We're not testing that the underlying math is right, those tests are in Apache Commons math, we're just testing the integration is working.

 

 


was (Author: joel.bernstein):
I'll fix this. I went overboard on these tests. No need to test that random behavior is as expected. Just need to test the function works and it brings back results that look right. We're not testing the underlying math is right, those tests are in Apache Commons math, we're just testing the integration is working.

 

 

> MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-13864
>                 URL: https://issues.apache.org/jira/browse/SOLR-13864
>             Project: Solr
>          Issue Type: Test
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Chris M. Hostetter
>            Assignee: Joel Bernstein
>            Priority: Major
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various jenkins boxes going back quite a while ... mostly from testGammaDistribution, but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
>  # tests that make assumptions about the relative values that will come out of taking samples from different random distributions that aren't garunteed to be true
>  ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and expecting one to always be strictly greater then the other. I'm not a stats guy, but my naive understanding is that on the low end some of these shapes may cross over, so every possible random sample from one shape is not garunteed to be less then every ossible random sample from a diff shape
>  # the code being tested does it's own randomization outside of the crontrol of the test framework (or test client)
>  ** this causes the seeds to not reproduce
> ----
> Tests should not be making assertions about random data that aren't 100% garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D + (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) < (4.99999D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for being controlled by the client/tests ... either via a request param used to initialize any new Random instances, or for example the use of the "tests.seed" property in various places in the code to try and provide some reproducibility even when the external solr client isn't even aware of randomization being a factor in the behavior of the code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org