You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by praveendareddy21 <gi...@git.apache.org> on 2016/05/22 05:46:58 UTC
[GitHub] spark pull request: [SPARK-15194] [ML] Add Python ML API for Multi...
GitHub user praveendareddy21 opened a pull request:
https://github.com/apache/spark/pull/13248
[SPARK-15194] [ML] Add Python ML API for MultivariateGaussian
## What changes were proposed in this pull request?
Added MultivariateGaussian in pyspark ML to match scala's ML API
## How was this patch tested?
Tested locally and also added testcases from scala's testsuite
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/praveendareddy21/spark local_branch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13248.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13248
----
commit a7250b4dd538be255f8220de20277d69edbeebac
Author: red <pr...@gmail.com>
Date: 2016-05-22T05:22:05Z
added Multivariate gaussian in ML Pyspark
commit 0c58e8866498d4e42af0542819fca8a6d76af08a
Author: red <pr...@gmail.com>
Date: 2016-05-22T05:33:56Z
added testcase for python multivariate
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by praveendareddy21 <gi...@git.apache.org>.
Github user praveendareddy21 commented on the issue:
https://github.com/apache/spark/pull/13248
@MechCoder @drcrallen @jjthomas @vectorijk
Kindly, review my new changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on the issue:
https://github.com/apache/spark/pull/13248
@praveendareddy21 Just made a first pass. Also please run PEP8 on your code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70182979
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
--- End diff --
`Vectors` is not imported in this module.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65798890
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
--- End diff --
you could use
```
`<http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case>`_
```
to make sure link displayed correctly in documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on the issue:
https://github.com/apache/spark/pull/13248
@praveendareddy21 For generating documentation for this API correctly, you could include this in `spark/python/docs/pyspark.ml.rst`
```
pyspark.ml.stat module
----------------------------
.. automodule:: pyspark.ml.stat
:members:
:undoc-members:
:inherited-members:
```
Also, just like @MechCoder said, you could run `spark/dev/lint-python` to make sure you pass all PEP8 checking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by praveendareddy21 <gi...@git.apache.org>.
Github user praveendareddy21 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70192124
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
--- End diff --
Vectors is used for creating Vector Object. But this class takes already created object as input.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70183009
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
+ s_pinv = self.__pinv_1d(s, eps)
+
+ # prec_U ndarray
+ # A decomposition such that np.dot(prec_U, prec_U.T)
+ # is the precision matrix, i.e. inverse of the covariance matrix.
+ self.prec_U = np.multiply(u, np.sqrt(s_pinv))
+
+ #log_det_cov : float
+ #Logarithm of the determinant of the covariance matrix
+ self.log_det_cov = np.sum(np.log(s[s > eps]))
+
+ except ValueError :
+ raise ValueError("Covariance matrix has no non-zero singular values")
+
+ def __pdf(self,x):
+ """
+ Calculates density at point x using precomputed Constants
+ """
+ return np.exp(self.__logpdf(x))
+
+ def __logpdf(self,x) :
+ """
+ Calculates log-density at point x using precomputed Constants
+
+ x Points at which to evaluate the log of the probability
+ density function
+ log_det_cov : float
+ Logarithm of the determinant of the covariance matrix
+
+ prec_U ndarray
+ A decomposition such that np.dot(prec_U, prec_U.T)
+ is the precision matrix, i.e. inverse of the covariance matrix.
+
--- End diff --
The docstring is inconsistent with the method signature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794325
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
--- End diff --
Also not sure if a one-liner deserves its own helper method with documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15194] [ML] Add Python ML API for Multi...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13248#issuecomment-220815663
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70183088
--- Diff: python/pyspark/ml/tests.py ---
@@ -1493,7 +1495,62 @@ def test_infer_schema(self):
self.assertTrue(m, self.sm1)
else:
raise ValueError("Expected a matrix but got type %r" % type(m))
+class MultiVariateGaussianTests(PySparkTestCase):
+ def test_univariate(self) :
+ x1=Vectors.dense([0.0])
+ x2=Vectors.dense([1.5])
+
+ mu = Vectors.dense([0.0])
+ sigma1= DenseMatrix(1, 1, [1.0])
+ dist1= MultivariateGaussian(mu, sigma1)
+
+ self.assertAlmostEqual(dist1.pdf(x1),0.39894, 5)
+ self.assertAlmostEqual(dist1.pdf(x2),0.12952, 5)
+
+ sigma2= DenseMatrix(1, 1, [4.0])
+ dist2= MultivariateGaussian(mu, sigma2)
+
+ self.assertAlmostEqual(dist2.pdf(x1),0.19947, 5)
+ self.assertAlmostEqual(dist2.pdf(x2),0.15057, 5)
+
+ def test_multivariate(self) :
+ x1=Vectors.dense([0.0, 0.0])
+ x2=Vectors.dense([1.0, 1.0])
+
+ mu = Vectors.dense([0.0, 0.0])
+ sigma1= DenseMatrix(2, 2, [1.0, 0.0, 0.0, 1.0])
+ dist1= MultivariateGaussian(mu, sigma1)
+
+ self.assertAlmostEqual(dist1.pdf(x1),0.159154, 5)
+ self.assertAlmostEqual(dist1.pdf(x2),0.05855, 5)
+
+ sigma2= DenseMatrix(2, 2, [4.0, -1.0, -1.0, 2.0])
+ dist2= MultivariateGaussian(mu, sigma2)
+
+ self.assertAlmostEqual(dist2.pdf(x1),0.060155, 5)
+ self.assertAlmostEqual(dist2.pdf(x2),0.0339717, 5)
+
+ def test_multivariate_degenerate(self) :
+ x1=Vectors.dense([0.0, 0.0])
+ x2=Vectors.dense([1.0, 1.0])
+
+ mu = Vectors.dense([0.0, 0.0])
+ sigma1= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ dist1= MultivariateGaussian(mu, sigma1)
+
+ self.assertAlmostEqual(dist1.pdf(x1),0.11254, 5)
+ self.assertAlmostEqual(dist1.pdf(x2),0.068259, 5)
+
+ def test_SPARK_11302(self) :
+ x=Vectors.dense([629, 640, 1.7188, 618.19])
+ mu = Vectors.dense([1055.3910505836575, 1070.489299610895, 1.39020554474708, 1040.5907503867697])
+ sigma= DenseMatrix(4, 4, [166769.00466698944, 169336.6705268059, 12.820670788921873, 164243.93314092053,
+ 169336.6705268059, 172041.5670061245, 21.62590020524533, 166678.01075856484,
+ 12.820670788921873, 21.62590020524533, 0.872524191943962, 4.283255814732373,
+ 164243.93314092053, 166678.01075856484, 4.283255814732373, 161848.9196719207])
+ dist= MultivariateGaussian(mu, sigma)
--- End diff --
Please format the code better (e.g. two elements in each row).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by praveendareddy21 <gi...@git.apache.org>.
Github user praveendareddy21 commented on the issue:
https://github.com/apache/spark/pull/13248
@vectorijk I will be pushing the changes in few days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on the issue:
https://github.com/apache/spark/pull/13248
Can you please reopen the pull request across the spark master branch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794904
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
+ s_pinv = self.__pinv_1d(s, eps)
+
+ # prec_U ndarray
+ # A decomposition such that np.dot(prec_U, prec_U.T)
+ # is the precision matrix, i.e. inverse of the covariance matrix.
+ self.prec_U = np.multiply(u, np.sqrt(s_pinv))
+
+ #log_det_cov : float
+ #Logarithm of the determinant of the covariance matrix
+ self.log_det_cov = np.sum(np.log(s[s > eps]))
+
+ except ValueError :
+ raise ValueError("Covariance matrix has no non-zero singular values")
+
+ def __pdf(self,x):
+ """
+ Calculates density at point x using precomputed Constants
+ """
+ return np.exp(self.__logpdf(x))
+
+ def __logpdf(self,x) :
+ """
+ Calculates log-density at point x using precomputed Constants
+
+ x Points at which to evaluate the log of the probability
+ density function
+ log_det_cov : float
+ Logarithm of the determinant of the covariance matrix
+
+ prec_U ndarray
+ A decomposition such that np.dot(prec_U, prec_U.T)
+ is the precision matrix, i.e. inverse of the covariance matrix.
+
+ """
+ x = self.__process_quantiles(x, self.dim)
+ dim = x.shape[-1]
+ delta = x - self.mu
+ maha = np.sum(np.square(np.dot(delta, self.prec_U)), axis=-1)
--- End diff --
you should replace `np.sum(np.square(` with `np.dot` operations.
delta_U = np.dot(delta, self.prec_U)
maha = np.dot(delta_U, delta_U)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794293
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
--- End diff --
# compute the pseudoinverse of s
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70182998
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
--- End diff --
Two many blank lines above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65793865
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
--- End diff --
This import should be moved above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794427
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
+ s_pinv = self.__pinv_1d(s, eps)
+
+ # prec_U ndarray
+ # A decomposition such that np.dot(prec_U, prec_U.T)
+ # is the precision matrix, i.e. inverse of the covariance matrix.
+ self.prec_U = np.multiply(u, np.sqrt(s_pinv))
+
+ #log_det_cov : float
+ #Logarithm of the determinant of the covariance matrix
+ self.log_det_cov = np.sum(np.log(s[s > eps]))
+
+ except ValueError :
+ raise ValueError("Covariance matrix has no non-zero singular values")
+
+ def __pdf(self,x):
+ """
+ Calculates density at point x using precomputed Constants
+ """
+ return np.exp(self.__logpdf(x))
+
+ def __logpdf(self,x) :
+ """
+ Calculates log-density at point x using precomputed Constants
+
+ x Points at which to evaluate the log of the probability
+ density function
+ log_det_cov : float
+ Logarithm of the determinant of the covariance matrix
+
+ prec_U ndarray
+ A decomposition such that np.dot(prec_U, prec_U.T)
+ is the precision matrix, i.e. inverse of the covariance matrix.
+
+ """
+ x = self.__process_quantiles(x, self.dim)
+ dim = x.shape[-1]
+ delta = x - self.mu
+ maha = np.sum(np.square(np.dot(delta, self.prec_U)), axis=-1)
+ return -0.5 * (dim * np.log(2 * np.pi) + self.log_det_cov + maha)
+
+
+
+
+ def __process_parameters(self, dim, mean, cov):
+ """
+ Helper funtion to process input values, based on scipy multivariate
+
+ Infer dimensionality from mean or covariance matrix, ensure that
+ mean and covariance are full vector resp. matrix.
+
+ """
+
+ # Try to infer dimensionality
+ if dim is None:
+ if mean is None:
+ if cov is None:
+ dim = 1
+ else:
+ cov = np.asarray(cov, dtype=float)
+ if cov.ndim < 2:
+ dim = 1
+ else:
+ dim = cov.shape[0]
+ else:
+ mean = np.asarray(mean, dtype=float)
+ dim = mean.size
+ else:
+ if not np.isscalar(dim):
+ raise ValueError("Dimension of random variable must be a scalar.")
+
+ # Check input sizes and return full arrays for mean and cov if necessary
+ if mean is None:
+ mean = np.zeros(dim)
+ mean = np.asarray(mean, dtype=float)
+
+ if cov is None:
+ cov = 1.0
+ cov = np.asarray(cov, dtype=float)
+
+ if dim == 1:
+ mean.shape = (1,)
+ cov.shape = (1, 1)
+
+ if mean.ndim != 1 or mean.shape[0] != dim:
+ raise ValueError("Array 'mean' must be vector of length %d." % dim)
+ if cov.ndim == 0:
+ cov = cov * np.eye(dim)
+ elif cov.ndim == 1:
+ cov = np.diag(cov)
+ else:
+ if cov.shape != (dim, dim):
+ raise ValueError("Array 'cov' must be at most two-dimensional,"
+ " but cov.ndim = %d" % cov.ndim)
+
+ return dim, mean, cov
+
+ def __process_quantiles(self, x, dim):
+ """
+ Helper funtion to process quantiles, based on scipy multivariate
+
+ Adjust quantiles array so that last axis labels the components of
+ each data point.
+
+ """
+ x = np.asarray(x, dtype=float)
+
+ if x.ndim == 0:
+ x = x[np.newaxis]
+ elif x.ndim == 1:
+ if dim == 1:
+ x = x[:, np.newaxis]
+ else:
+ x = x[np.newaxis, :]
+
+ return x
+
+
+ def __pinv_1d(self, v, eps=1e-5):
+ """
+ A helper function for computing the pseudoinverse, based on scipy multivariate
+
+ v : iterable of numbers
+ This may be thought of as a vector of eigenvalues or singular values.
+ eps : float
+ Elements of v smaller than eps are considered negligible.
+ returns 1d float ndarray
+ A vector of pseudo-inverted numbers.
+ """
+ return np.array([0 if abs(x) < eps else 1/x for x in v], dtype=float)
--- End diff --
This can be easily vectorized. You are creating a copy anyway.
v_c = v.copy()
mask = v_c <= eps
v_c[mask] = 0.0
v_c[~mask] = 1. / v_c[~mask]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794056
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
+ s_pinv = self.__pinv_1d(s, eps)
+
+ # prec_U ndarray
+ # A decomposition such that np.dot(prec_U, prec_U.T)
+ # is the precision matrix, i.e. inverse of the covariance matrix.
+ self.prec_U = np.multiply(u, np.sqrt(s_pinv))
+
+ #log_det_cov : float
+ #Logarithm of the determinant of the covariance matrix
+ self.log_det_cov = np.sum(np.log(s[s > eps]))
+
+ except ValueError :
+ raise ValueError("Covariance matrix has no non-zero singular values")
+
+ def __pdf(self,x):
+ """
+ Calculates density at point x using precomputed Constants
+ """
+ return np.exp(self.__logpdf(x))
+
+ def __logpdf(self,x) :
+ """
+ Calculates log-density at point x using precomputed Constants
+
+ x Points at which to evaluate the log of the probability
+ density function
+ log_det_cov : float
+ Logarithm of the determinant of the covariance matrix
+
+ prec_U ndarray
+ A decomposition such that np.dot(prec_U, prec_U.T)
+ is the precision matrix, i.e. inverse of the covariance matrix.
+
+ """
+ x = self.__process_quantiles(x, self.dim)
+ dim = x.shape[-1]
+ delta = x - self.mu
+ maha = np.sum(np.square(np.dot(delta, self.prec_U)), axis=-1)
+ return -0.5 * (dim * np.log(2 * np.pi) + self.log_det_cov + maha)
+
+
+
+
+ def __process_parameters(self, dim, mean, cov):
--- End diff --
This method is redundant right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70183002
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
--- End diff --
Insert one space before and after the `=`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65793951
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
--- End diff --
You can use the `numRows`, `numCols`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/13248
Can you close this one then?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65798896
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
--- End diff --
same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by vectorijk <gi...@git.apache.org>.
Github user vectorijk commented on the issue:
https://github.com/apache/spark/pull/13248
ping @praveendareddy21 Is this still active? If not, I could help with this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794126
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
--- End diff --
Should we fall back to SciPy's multivariate normal if that is present?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13248
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794779
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
+ s_pinv = self.__pinv_1d(s, eps)
+
+ # prec_U ndarray
+ # A decomposition such that np.dot(prec_U, prec_U.T)
+ # is the precision matrix, i.e. inverse of the covariance matrix.
+ self.prec_U = np.multiply(u, np.sqrt(s_pinv))
--- End diff --
`u * np.sqrt(s_pinv)` should be a shorthand to do the same thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13248: [SPARK-15194] [ML] Add Python ML API for MultivariateGau...
Posted by praveendareddy21 <gi...@git.apache.org>.
Github user praveendareddy21 commented on the issue:
https://github.com/apache/spark/pull/13248
Reopened the pull request on master branch.
https://github.com/apache/spark/pull/14375
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by MechCoder <gi...@git.apache.org>.
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r65794836
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
+
+ """
+
+ def __init__(self, mu, sigma):
+ """
+ __init__(self, mu, sigma)
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+ mu and sigma must be instances of DenseVector and DenseMatrix respectively.
+
+ """
+
+
+ assert (isinstance(mu, DenseVector)), "mu must be a DenseVector Object"
+ assert (isinstance(sigma, DenseMatrix)), "sigma must be a DenseMatrix Object"
+
+ sigma_shape=sigma.toArray().shape
+ assert (sigma_shape[0]==sigma_shape[1]) , "Covariance matrix must be square"
+ assert (sigma_shape[0]==mu.size) , "Mean vector length must match covariance matrix size"
+
+ # initialize eagerly precomputed attributes
+
+ self.mu=mu
+
+ # storing sigma as numpy.ndarray
+ # furthur calculations are done ndarray only
+ self.sigma=sigma.toArray()
+
+
+ # initialize attributes to be computed later
+
+ self.prec_U = None
+ self.log_det_cov = None
+
+ # compute distribution dependent constants
+ self.__calculateCovarianceConstants()
+
+
+ def pdf(self,x):
+ """
+ Returns density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__pdf(x))
+
+ def logpdf(self,x):
+ """
+ Returns the log-density of this multivariate Gaussian at a point given by Vector x
+ """
+ assert (isinstance(x, Vector)), "x must be of Vector Type"
+ return float(self.__logpdf(x))
+
+ def __calculateCovarianceConstants(self):
+ """
+ Calculates distribution dependent components used for the density function
+ based on scipy multivariate library
+ refer https://github.com/scipy/scipy/blob/master/scipy/stats/_multivariate.py
+ tested with precision of 9 significant digits(refer testcase)
+
+
+ """
+
+ try :
+ # pre-processing input parameters
+ # throws ValueError with invalid inputs
+ self.dim, self.mu, self.sigma = self.__process_parameters(None, self.mu, self.sigma)
+
+ # return the eigenvalues and eigenvectors
+ # of a Hermitian or symmetric matrix.
+ # s = eigen values
+ # u = eigen vectors
+ s, u = np.linalg.eigh(self.sigma)
+
+ #Singular values are considered to be non-zero only if
+ #they exceed a tolerance based on machine precision, matrix size, and
+ #relation to the maximum singular value (same tolerance used by, e.g., Octave).
+
+ # calculation for machine precision
+ t = u.dtype.char.lower()
+ factor = {'f': 1E3, 'd': 1E6}
+ cond = factor[t] * np.finfo(t).eps
+
+ eps = cond * np.max(abs(s))
+
+ # checkng whether covariance matrix has any non-zero singular values
+ if np.min(s) < -eps:
+ raise ValueError
+
+ #computing the pseudoinverse
+ s_pinv = self.__pinv_1d(s, eps)
+
+ # prec_U ndarray
+ # A decomposition such that np.dot(prec_U, prec_U.T)
+ # is the precision matrix, i.e. inverse of the covariance matrix.
+ self.prec_U = np.multiply(u, np.sqrt(s_pinv))
+
+ #log_det_cov : float
+ #Logarithm of the determinant of the covariance matrix
+ self.log_det_cov = np.sum(np.log(s[s > eps]))
+
+ except ValueError :
+ raise ValueError("Covariance matrix has no non-zero singular values")
+
+ def __pdf(self,x):
+ """
+ Calculates density at point x using precomputed Constants
+ """
+ return np.exp(self.__logpdf(x))
+
+ def __logpdf(self,x) :
+ """
+ Calculates log-density at point x using precomputed Constants
+
+ x Points at which to evaluate the log of the probability
+ density function
+ log_det_cov : float
+ Logarithm of the determinant of the covariance matrix
+
+ prec_U ndarray
+ A decomposition such that np.dot(prec_U, prec_U.T)
+ is the precision matrix, i.e. inverse of the covariance matrix.
+
+ """
+ x = self.__process_quantiles(x, self.dim)
--- End diff --
It is safe to assume x is 1-D. You have just asserted that it is a Vector above. (i.e do away with this)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70198232
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
--- End diff --
I see, but the missing import of `Vectors` would fail the doctest.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13248: [SPARK-15194] [ML] Add Python ML API for Multivar...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13248#discussion_r70182969
--- Diff: python/pyspark/ml/stat/distribution.py ---
@@ -0,0 +1,267 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.ml.linalg import DenseVector, DenseMatrix, Vector
+import numpy as np
+
+__all__ = ['MultivariateGaussian']
+
+
+
+class MultivariateGaussian():
+ """
+ This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution. In
+ the event that the covariance matrix is singular, the density will be computed in a
+ reduced dimensional subspace under which the distribution is supported.
+ (see [[http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Degenerate_case]])
+
+ mu The mean vector of the distribution
+ sigma The covariance matrix of the distribution
+
+
+ >>> mu = Vectors.dense([0.0, 0.0])
+ >>> sigma= DenseMatrix(2, 2, [1.0, 1.0, 1.0, 1.0])
+ >>> x = Vectors.dense([1.0, 1.0])
+ >>> m = MultivariateGaussian(mu, sigma)
+ >>> m.pdf(x)
+ 0.0682586811486
--- End diff --
To run the doctest, I think we need to call the `doctest.testmod()` explicitly like other modules do. Check [mllib/util.py](https://github.com/apache/spark/blob/v2.0.0-rc2/python/pyspark/mllib/util.py#L509-L528).
Also need to add this module to the `python_test_goals` to pyspark_ml module object of [dev/sparktestsupport/modules.py](https://github.com/apache/spark/blob/v2.0.0-rc2/dev/sparktestsupport/modules.py#L401-L411)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org