[jira] [Comment Edited] (SPARK-2599) almostEquals mllib.util.TestingUtils does not behave as expected when comparing against 0.0

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067876#comment-14067876 ] 

Sean Owen edited comment on SPARK-2599 at 7/20/14 10:20 AM:
------------------------------------------------------------

Yes, the relative error will never be more than 2.0; it wouldn't be possible to use epsilon >= 2.0 meaningfully -- think of expecting 0.05 but accepting anything 0.05 +/- 0.1. The range is also asymmetric, which might not be desired. With expected value 2.0 and epsilon = 0.1, you allow the range 1.8 to 2.2222 instead of 1.8 to 2.2.

The usual approach is to treat epsilon as an absolute error. This is fit for purpose in its default role of ignoring variance due to rounding or machine precision. The default can be near a ulp, and can be changed.

Is the intent really to make an *absolute* error bound whose default value is some *relative proportion* of the absolute value of the arguments?

If relative error, then it should be relative to the absolute value of the expected value, rather than just larger one, I think.

Of course, you would still be able to change the value of this relative epsilon. But what do you do when the expected value is 0? you would have no way of specifying an epsilon other than 0. And, working in relative terms might get clunky; if the expected value is 0.123 +/ 0.45, the test writer has to work out and document what the relative epsilon is doing.

Still, relative is useful. It seems like the right-est thing is two methods, one specifying absolute and the other relative complexity. And that is a little complex.

So I can see why every testing framework I've seen just lets you set an absolute error and that's it. At the least, it's straightforward to reason about and fully flexible. You do have to set the epsilon once in a while, but that's always the case.

Net-net, I'm suggesting making this an absolute error only to address all this.


was (Author: srowen):
The relative error will never be more than 2.0; it wouldn't be possible to use epsilon >= 2.0 meaningfully -- think of expecting 0.005 but accepting anything 0.05 +/- 0.1. The range is also asymmetric, which might not be desired. With expected value 2.0 and epsilon = 0.1, you allow the range 1.8 to 2.2222 instead of 1.8 to 2.2.

The usual approach is to treat epsilon as an absolute error. This is fit for purpose in its default role of ignoring variance due to rounding or machine precision. The default can be near a ulp, and can be changed.

Is the intent really to make an *absolute* error bound whose default value is some *relative proportion* of the absolute value of the arguments?

If relative error, then it should be relative to the absolute value of the expected value, rather than just larger one, I think.

Of course, you would still be able to change the value of this relative epsilon. But what do you do when the expected value is 0? you would have no way of specifying an epsilon other than 0. And, working in relative terms might get clunky; if the expected value is 0.123 +/ 0.45, the test writer has to work out and document what the relative epsilon is doing.

Still, relative is useful. It seems like the right-est thing is two methods, one specifying absolute and the other relative complexity. And that is a little complex.

So I can see why every testing framework I've seen just lets you set an absolute error and that's it. At the least, it's straightforward to reason about and fully flexible. You do have to set the epsilon once in a while, but that's always the case.

Net-net, I'm suggesting making this an absolute error only to address all this.

> almostEquals mllib.util.TestingUtils does not behave as expected when comparing against 0.0
> -------------------------------------------------------------------------------------------
>
>                 Key: SPARK-2599
>                 URL: https://issues.apache.org/jira/browse/SPARK-2599
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>            Reporter: Doris Xin
>            Priority: Minor
>
> DoubleWithAlmostEquals.almostEquals, when used to compare a number with 0.0, would always produce an epsilon of 1 >> 1e-10, causing false failure when comparing very small numbers with 0.0.



--
This message was sent by Atlassian JIRA
(v6.2#6252)