You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jeff Eastman (Created) (JIRA)" <ji...@apache.org> on 2011/10/19 19:07:10 UTC
[jira] [Created] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Improve Scalability of Gaussian Cluster For Wide Vectors
--------------------------------------------------------
Key: MAHOUT-846
URL: https://issues.apache.org/jira/browse/MAHOUT-846
Project: Mahout
Issue Type: Improvement
Affects Versions: 0.5
Reporter: Jeff Eastman
Assignee: Jeff Eastman
Fix For: 0.6
The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
public double pdf(VectorWritable vw) {
Vector x = vw.get();
// return the product of the component pdfs
// TODO: is this reasonable? correct? It seems to work in some cases.
double pdf = 1;
for (int i = 0; i < x.size(); i++) {
// small prior on stdDev to avoid numeric instability when stdDev==0
pdf *= UncommonDistributions.dNorm(x.getQuick(i),
getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
}
return pdf;
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131309#comment-13131309 ]
Hudson commented on MAHOUT-846:
-------------------------------
Integrated in Mahout-Quality #1105 (See [https://builds.apache.org/job/Mahout-Quality/1105/])
MAHOUT-846: Simplified DistanceMeasureCluster.pdf() to avoid exp(...). Increased DPC iterations to 20 in build-reuters. All tests run
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186540
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/DistanceMeasureCluster.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/TestClusterClassifier.java
* /mahout/trunk/examples/bin/build-reuters.sh
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAHOUT-846) Improve Scalability of
Gaussian Cluster For Wide Vectors
Posted by "Jeff Eastman (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174953#comment-13174953 ]
Jeff Eastman edited comment on MAHOUT-846 at 12/22/11 6:43 PM:
---------------------------------------------------------------
The following code refactors the pdf() function to sum the exponents and perform the exponentiation only once. It produces equivalent results to the old version for DisplayDirichlet and the math seems solid. I introduced two helper methods to make it all clearer. I believe it will handle wide vectors better than the original but have yet to verify that.
{code}
@Override
public double pdf(VectorWritable vw) {
Vector x = vw.get();
Vector m = getCenter();
Vector s = getRadius().plus(0.0000001); // add a small prior to avoid divide
// by zero
return Math.exp(-(divideSquareAndSum(x.minus(m), s) / 2))
/ zProd(s.times(UncommonDistributions.SQRT2PI));
}
private double zProd(Vector s) {
double prod = 1;
for (int i = 0; i < s.size(); i++) {
prod *= s.getQuick(i);
}
return prod;
}
private double divideSquareAndSum(Vector numerator, Vector denominator) {
double result = 0;
for (Iterator<Element> it = denominator.iterateNonZero(); it.hasNext();) {
Element denom = it.next();
double quotient = numerator.getQuick(denom.index()) / denom.get();
result += quotient * quotient;
}
return result;
}
{code}
was (Author: jeastman):
The following code refactors the pdf() function to sum the exponents and perform the exponentiation only once. It produces equivalent results to the old version for DisplayDirichlet and the math seems solid. I introduced two helper methods to make it all clearer. I believe it will handle wide vectors better than the original but have yet to verify that.
{code}
public double pdf(VectorWritable vw) {
Vector x = vw.get();
Vector m = getCenter();
Vector s = getRadius().plus(0.0000001); // add a small prior to avoid divide by zero
Vector xms2 = divideAndSquare(x.minus(m), s);
return Math.exp(-(xms2.zSum() / 2))
/ zProd(s.times(UncommonDistributions.SQRT2PI));
}
private double zProd(Vector s) {
double prod = 1;
for (int i = 0; i < s.size(); i++) {
prod *= s.getQuick(i);
}
return prod;
}
private Vector divideAndSquare(Vector numerator, Vector denominator) {
Vector result = numerator.like();
for (Iterator<Element> it = denominator.iterateNonZero(); it.hasNext();) {
Element denom = it.next();
double quotient = numerator.getQuick(denom.index()) / denom.get();
result.setQuick(denom.index(), quotient * quotient);
}
return result;
}
{code}
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175255#comment-13175255 ]
Hudson commented on MAHOUT-846:
-------------------------------
Integrated in Mahout-Quality #1265 (See [https://builds.apache.org/job/Mahout-Quality/1265/])
MAHOUT-846: Minor refactoring to eliminate unnecessary vector.times(SQRT2PI)
MAHOUT-846: Refactored pdf() to sum the exponents and perform the exponentiation only once. All tests run
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222443
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/models/GaussianCluster.java
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222420
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/UncommonDistributions.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/models/GaussianCluster.java
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175954#comment-13175954 ]
Hudson commented on MAHOUT-846:
-------------------------------
Integrated in Mahout-Quality #1272 (See [https://builds.apache.org/job/Mahout-Quality/1272/])
MAHOUT-846: Improved scalability of GaussianCluster.pdf. Introduced some beginnings for MAHOUT-933. All tests run.
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224730
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/AbstractCluster.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/CIMapper.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/CIReducer.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/Cluster.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/ClusterClassifier.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/ClusterIterator.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/Model.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/DirichletCluster.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/models/GaussianCluster.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/TestClusterClassifier.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayDirichlet.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayFuzzyKMeans.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayKMeans.java
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Jeff Eastman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174953#comment-13174953 ]
Jeff Eastman commented on MAHOUT-846:
-------------------------------------
The following code refactors the pdf() function to sum the exponents and perform the exponentiation only once. It produces equivalent results to the old version for DisplayDirichlet and the math seems solid. I introduced two helper methods to make it all clearer. I believe it will handle wide vectors better than the original but have yet to verify that.
{code}
public double pdf(VectorWritable vw) {
Vector x = vw.get();
Vector m = getCenter();
Vector s = getRadius().plus(0.0000001); // add a small prior to avoid divide by zero
Vector xms2 = divideAndSquare(x.minus(m), s);
return Math.exp(-(xms2.zSum() / 2))
/ zProd(s.times(UncommonDistributions.SQRT2PI));
}
private double zProd(Vector s) {
double prod = 1;
for (int i = 0; i < s.size(); i++) {
prod *= s.getQuick(i);
}
return prod;
}
private Vector divideAndSquare(Vector numerator, Vector denominator) {
Vector result = numerator.like();
for (Iterator<Element> it = denominator.iterateNonZero(); it.hasNext();) {
Element denom = it.next();
double quotient = numerator.getQuick(denom.index()) / denom.get();
result.setQuick(denom.index(), quotient * quotient);
}
return result;
}
{code}
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAHOUT-846) Improve Scalability of
Gaussian Cluster For Wide Vectors
Posted by "Jeff Eastman (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174953#comment-13174953 ]
Jeff Eastman edited comment on MAHOUT-846 at 12/22/11 8:41 PM:
---------------------------------------------------------------
The following code refactors the pdf() function to sum the exponents and perform the exponentiation only once. It produces equivalent results to the old version for DisplayDirichlet and the math seems solid. I introduced two helper methods to make it all clearer. I believe it will handle wide vectors better than the original but have yet to verify that.
{code}
@Override
public double pdf(VectorWritable vw) {
Vector x = vw.get();
Vector m = getCenter();
Vector s = getRadius().plus(0.0000001); // add a small prior to avoid divide by zero
return Math.exp(-(divideSquareAndSum(x.minus(m), s) / 2)) / zProdSqt2Pi(s);
}
private double zProdSqt2Pi(Vector s) {
double prod = 1;
for (int i = 0; i < s.size(); i++) {
prod *= s.getQuick(i) * UncommonDistributions.SQRT2PI;
}
return prod;
}
private double divideSquareAndSum(Vector numerator, Vector denominator) {
double result = 0;
for (Iterator<Element> it = denominator.iterateNonZero(); it.hasNext();) {
Element denom = it.next();
double quotient = numerator.getQuick(denom.index()) / denom.get();
result += quotient * quotient;
}
return result;
}
{code}
was (Author: jeastman):
The following code refactors the pdf() function to sum the exponents and perform the exponentiation only once. It produces equivalent results to the old version for DisplayDirichlet and the math seems solid. I introduced two helper methods to make it all clearer. I believe it will handle wide vectors better than the original but have yet to verify that.
{code}
@Override
public double pdf(VectorWritable vw) {
Vector x = vw.get();
Vector m = getCenter();
Vector s = getRadius().plus(0.0000001); // add a small prior to avoid divide
// by zero
return Math.exp(-(divideSquareAndSum(x.minus(m), s) / 2))
/ zProd(s.times(UncommonDistributions.SQRT2PI));
}
private double zProd(Vector s) {
double prod = 1;
for (int i = 0; i < s.size(); i++) {
prod *= s.getQuick(i);
}
return prod;
}
private double divideSquareAndSum(Vector numerator, Vector denominator) {
double result = 0;
for (Iterator<Element> it = denominator.iterateNonZero(); it.hasNext();) {
Element denom = it.next();
double quotient = numerator.getQuick(denom.index()) / denom.get();
result += quotient * quotient;
}
return result;
}
{code}
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Jeff Eastman (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman resolved MAHOUT-846.
---------------------------------
Resolution: Fixed
Committed changes to GaussianCluster.pdf(). Overloaded setRadius() to calculate zProd2piR denominator vs lazy initialization in above patch. All tests run and pdf() is significantly faster with no new Vectors created.
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Jeff Eastman (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-846:
--------------------------------
Description:
The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
{noformat}
public double pdf(VectorWritable vw) {
Vector x = vw.get();
// return the product of the component pdfs
// TODO: is this reasonable? correct? It seems to work in some cases.
double pdf = 1;
for (int i = 0; i < x.size(); i++) {
// small prior on stdDev to avoid numeric instability when stdDev==0
pdf *= UncommonDistributions.dNorm(x.getQuick(i),
getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
}
return pdf;
{noformat}
}
was:
The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
public double pdf(VectorWritable vw) {
Vector x = vw.get();
// return the product of the component pdfs
// TODO: is this reasonable? correct? It seems to work in some cases.
double pdf = 1;
for (int i = 0; i < x.size(); i++) {
// small prior on stdDev to avoid numeric instability when stdDev==0
pdf *= UncommonDistributions.dNorm(x.getQuick(i),
getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
}
return pdf;
}
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175277#comment-13175277 ]
Hudson commented on MAHOUT-846:
-------------------------------
Integrated in Mahout-Quality #1266 (See [https://builds.apache.org/job/Mahout-Quality/1266/])
MAHOUT-846: Cache pdf zProd2piR term constant over life of cluster
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1222524
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/models/GaussianCluster.java
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131194#comment-13131194 ]
Hudson commented on MAHOUT-846:
-------------------------------
Integrated in Mahout-Quality #1104 (See [https://builds.apache.org/job/Mahout-Quality/1104/])
MAHOUT-846: Fixed prior of DMCD to have Gaussian element values vs. all 0. Updated build-reuters.sh to use DMCD. All tests run and reuters looks much better.
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186452
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/dirichlet/models/DistanceMeasureClusterDistribution.java
* /mahout/trunk/examples/bin/build-reuters.sh
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-846) Improve Scalability of Gaussian
Cluster For Wide Vectors
Posted by "Jeff Eastman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175189#comment-13175189 ]
Jeff Eastman commented on MAHOUT-846:
-------------------------------------
This version gets rid of all vector operations and caches the zProd2piS term which is constant over the life of the cluster between reCalculations. I don't really like the lazy evaluation but the alternative is putting it in the constructors and reader. Will see...
{code}
// the value of the zProduct(S*2pi) term. Calculated below.
private Double zProd2piS = null;
@Override
public double pdf(VectorWritable vw) {
if (zProd2piS == null) {
computeProd2piS();
}
return Math.exp(-(sumXminusMdivSsquared(vw.get()) / 2)) / zProd2piS;
}
/**
* Compute the product(s[i]*SQRT2PI) over all i
*/
private void computeProd2piS() {
zProd2piS = 1.0;
for (Iterator<Element> it = getRadius().iterateNonZero(); it.hasNext();) {
Element se = it.next();
zProd2piS *= se.get() * UncommonDistributions.SQRT2PI;
}
}
@Override
public void computeParameters() {
super.computeParameters();
zProd2piS = null;
}
/**
* @param x
* a Vector
* @return the zSum(((x[i]-m[i])/s[i])^2) over i
*/
private double sumXminusMdivSsquared(Vector x) {
double result = 0;
for (Iterator<Element> it = getRadius().iterateNonZero(); it.hasNext();) {
Element se = it.next();
double quotient = (x.get(se.index()) - getCenter().get(se.index()))
/ se.get();
result += quotient * quotient;
}
return result;
}
{code}
> Improve Scalability of Gaussian Cluster For Wide Vectors
> --------------------------------------------------------
>
> Key: MAHOUT-846
> URL: https://issues.apache.org/jira/browse/MAHOUT-846
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
>
> The pdf() implementation in GaussianCluster is pretty lame. It is computing a running product of the element pdfs which, for wide input vectors (Reuters is 41,807), always underflows and returns 0. Here's the code:
> {noformat}
> public double pdf(VectorWritable vw) {
> Vector x = vw.get();
> // return the product of the component pdfs
> // TODO: is this reasonable? correct? It seems to work in some cases.
> double pdf = 1;
> for (int i = 0; i < x.size(); i++) {
> // small prior on stdDev to avoid numeric instability when stdDev==0
> pdf *= UncommonDistributions.dNorm(x.getQuick(i),
> getCenter().getQuick(i), getRadius().getQuick(i) + 0.000001);
> }
> return pdf;
> {noformat}
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira