You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/11/23 14:56:00 UTC
[jira] [Assigned] (SPARK-26158) Enhance the accuracy of covariance
in RowMatrix for DenseVector
[ https://issues.apache.org/jira/browse/SPARK-26158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-26158:
------------------------------------
Assignee: (was: Apache Spark)
> Enhance the accuracy of covariance in RowMatrix for DenseVector
> ---------------------------------------------------------------
>
> Key: SPARK-26158
> URL: https://issues.apache.org/jira/browse/SPARK-26158
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.4.0
> Reporter: Liang Li
> Priority: Minor
>
> Compare Spark computeCovariance function in RowMatrix for DenseVector and Numpy's function cov,
> *Find two problem, below is the result:*
> *1)The Spark function computeCovariance in RowMatrix is not accuracy*
> input data
> 1.0,2.0,3.0,4.0,5.0
> 2.0,3.0,1.0,2.0,6.0
> Numpy function cov result:
> [[2.5 1.75]
> [ 1.75 3.7 ]]
> RowMatrix function computeCovariance result:
> 2.5 1.75
> 1.75 3.700000000000001
>
> 2)For some input case, the result is not good
> generate input data by below logic
> data1 = np.random.normal(loc=100000, scale=0.000009, size=10000000)
> data2 = np.random.normal(loc=200000, scale=0.000002,size=10000000)
>
> Numpy function cov result:
> [[ 8.10536442e-11 -4.35439574e-15]
> [ -4.35439574e-15 3.99928264e-12]]
>
> RowMatrix function computeCovariance result:
> -0.0027484893798828125 0.001491546630859375
> 0.001491546630859375 8.087158203125E-4
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org