You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Thomas Neidhart (JIRA)" <ji...@apache.org> on 2015/01/20 13:08:35 UTC
[jira] [Commented] (MATH-1197) Incorrect Kolmogorov–Smirnov Statistic for two samples
[ https://issues.apache.org/jira/browse/MATH-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14283756#comment-14283756 ]
Thomas Neidhart commented on MATH-1197:
---------------------------------------
One observation: the samples contain a lot of equal values.
The KS test statistic is implemented using Arrays.binarySearch, but this does not specify which index will be found when looking for a given value in a sorted array.
E.g. if you have samples [0, 0, 0, 0, 0, 1] and you search for 0, you might get an index in the range [0, 4]. As far as I understand the KS statistic, it is an empirical distribution function which calculates the cumulative density based on how many values are less or equal than the given observation, which is not equal to the result returned by Arrays.binarySearch.
> Incorrect Kolmogorov–Smirnov Statistic for two samples
> -------------------------------------------------------
>
> Key: MATH-1197
> URL: https://issues.apache.org/jira/browse/MATH-1197
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 3.4.1
> Environment: Ubuntu 14.04
> Reporter: Danaja Thiyunuwan Maldeniya
>
> kolmogorovSmirnovTest(double[],double[]) against the samples given below gives 5.699107852308316E-12 instead of 0.9793 (approx.) Traced the issue to kolmogorovSmirnovStatistic(double[],double[]) which gives 0.49507389162561577 instead of 0.064 (verified with ks.test in R and JDistlib)
> double[] x = {0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,2.202653,2.202653,2.202653
> ,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653
> ,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653
> ,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,3.181199,3.181199,3.181199,3.181199,3.181199,3.181199,3.723539
> ,3.723539,3.723539,3.723539,4.383482,4.383482,4.383482,4.383482,5.320671,5.320671,5.320671,5.717284,6.964001,7.352165
> ,8.710510,8.710510,8.710510,8.710510,8.710510,8.710510,9.539004,9.539004, 10.720619, 17.726077, 17.726077, 17.726077, 17.726077
> ,22.053875 ,23.799144 ,27.355308 ,30.584960 ,30.584960 ,30.584960, 30.584960, 30.751808};
> double[] y = {0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000
> ,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,0.000000,2.202653
> ,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,2.202653,3.061758,3.723539,5.628420,5.628420,5.628420,5.628420
> ,5.628420,6.916982,6.916982,6.916982, 10.178538, 10.178538, 10.178538, 10.178538, 10.178538 };
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)