[jira] [Commented] (SPARK-18581) MultivariateGaussian does not check if covariance matrix is invertible

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15695359#comment-15695359 ] 

Sean Owen commented on SPARK-18581:
-----------------------------------

It sounds like there's definitely a problem here but why is the pseudo-determinant the issue? it's possible for eigenvalues to be in (0,1) in which case the sum of logs is negative. Is that a problem per se?

> MultivariateGaussian does not check if covariance matrix is invertible
> ----------------------------------------------------------------------
>
>                 Key: SPARK-18581
>                 URL: https://issues.apache.org/jira/browse/SPARK-18581
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.6.2, 2.0.2
>            Reporter: Hao Ren
>
> When training GaussianMixtureModel, I found some probability much larger than 1. That leads me to that fact that, the value returned by MultivariateGaussian.pdf can be 10^5, etc.
> After reviewing the code, I found that problem lies in the computation of determinant of the covariance matrix.
> The computation is simplified by using pseudo-determinant of a positive defined matrix. However, if the eigen value is all between 0 and 1, log(pseudo-determinant) will be a negative number like,  -50. As a result, the logpdf could be positive, thus pdf > 1
> The related code that the following:
> // In function: MultivariateGaussian.calculateCovarianceConstants
> {code}
> val logPseudoDetSigma = d.activeValuesIterator.filter(_ > tol).map(math.log).sum
> {code}
> d is the eigen value vector here. If lots of its elements are between 0 and 1, then logPseudoDetSigma could be negative.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org