[jira] [Commented] (FLINK-2832) Failing test: RandomSamplerTest.testReservoirSamplerWithReplacement

["Till Rohrmann (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316536#comment-15316536 ] 

Till Rohrmann commented on FLINK-2832:
--------------------------------------

Here is another instance of the failure (this time the {{testReservoirSamplerWithMultiSourcePartitions1}} test failed): https://s3.amazonaws.com/archive.travis-ci.org/jobs/134215943/log.txt

The problem seems to be that the testing condition is not deterministic. Every so and so many test runs the Kolmogorow-Smirnow test simply fails because the level of significance is not zero.

I would propose to set the level of significance to {{0.001}} instead of {{0.01}} and to add a retry annotation to the sampler tests. This should harden the test stability.

> Failing test: RandomSamplerTest.testReservoirSamplerWithReplacement
> -------------------------------------------------------------------
>
>                 Key: FLINK-2832
>                 URL: https://issues.apache.org/jira/browse/FLINK-2832
>             Project: Flink
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 0.10.0
>            Reporter: Vasia Kalavri
>            Priority: Critical
>              Labels: test-stability
>             Fix For: 1.0.0
>
>
> Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 19.133 sec <<< FAILURE! - in org.apache.flink.api.java.sampling.RandomSamplerTest
> testReservoirSamplerWithReplacement(org.apache.flink.api.java.sampling.RandomSamplerTest)  Time elapsed: 2.534 sec  <<< FAILURE!
> java.lang.AssertionError: KS test result with p value(0.110000), d value(0.103090)
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.assertTrue(Assert.java:41)
> 	at org.apache.flink.api.java.sampling.RandomSamplerTest.verifyKSTest(RandomSamplerTest.java:342)
> 	at org.apache.flink.api.java.sampling.RandomSamplerTest.verifyRandomSamplerWithSampleSize(RandomSamplerTest.java:330)
> 	at org.apache.flink.api.java.sampling.RandomSamplerTest.verifyReservoirSamplerWithReplacement(RandomSamplerTest.java:289)
> 	at org.apache.flink.api.java.sampling.RandomSamplerTest.testReservoirSamplerWithReplacement(RandomSamplerTest.java:192)
> Results :
> Failed tests: 
>   RandomSamplerTest.testReservoirSamplerWithReplacement:192->verifyReservoirSamplerWithReplacement:289->verifyRandomSamplerWithSampleSize:330->verifyKSTest:342 KS test result with p value(0.110000), d value(0.103090)
> Full log [here|https://travis-ci.org/apache/flink/jobs/84120131].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)