[jira] [Created] (SPARK-14053) Merge absTol and relTol into one in MLlib tests

["Xiangrui Meng (JIRA)" <ji...@apache.org>]

Xiangrui Meng created SPARK-14053:
-------------------------------------

             Summary: Merge absTol and relTol into one in MLlib tests
                 Key: SPARK-14053
                 URL: https://issues.apache.org/jira/browse/SPARK-14053
             Project: Spark
          Issue Type: Improvement
          Components: ML, MLlib, Tests
    Affects Versions: 2.0.0
            Reporter: Xiangrui Meng
            Assignee: Xiangrui Meng


We have absTol and relTol in MLlib tests to compare values with possible numerical differences. However, in most cases we should just use relTol. Many absTol are not used properly. See https://github.com/apache/spark/search?q=absTol. One corner case relTol doesn't handle is when the target value is 0. We can make the following change to relTol to solve the issue. Consider `x ~== y relTol t`.

1. If abs(y) > eps / t, test abs(y - x) / abs(y) < t,
2. else test abs(y - x) < eps

where eps is a reasonably small value, e.g., 1e-14. Note that the transition is smooth at abs(y) = eps / t.

cc [~dbtsai]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org