You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jeff Eastman (Created) (JIRA)" <ji...@apache.org> on 2011/09/27 00:07:12 UTC
[jira] [Created] (MAHOUT-818) Canopy Emits Too Many Trivial
Clusters
Canopy Emits Too Many Trivial Clusters
--------------------------------------
Key: MAHOUT-818
URL: https://issues.apache.org/jira/browse/MAHOUT-818
Project: Mahout
Issue Type: Improvement
Components: Clustering
Affects Versions: 0.5
Reporter: Jeff Eastman
Assignee: Jeff Eastman
Fix For: 0.6
Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, -cf (--clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-818) Canopy Emits Too Many Trivial
Clusters
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116041#comment-13116041 ]
Hudson commented on MAHOUT-818:
-------------------------------
Integrated in Mahout-Quality #1066 (See [https://builds.apache.org/job/Mahout-Quality/1066/])
MAHOUT-818: Patch to filter canopies based upon numPoints. All tests run
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176603
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyConfigKeys.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyDriver.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyMapper.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/canopy/CanopyReducer.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/commandline/DefaultOptionCreator.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/canopy/TestCanopyCreation.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayCanopy.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayFuzzyKMeans.java
> Canopy Emits Too Many Trivial Clusters
> --------------------------------------
>
> Key: MAHOUT-818
> URL: https://issues.apache.org/jira/browse/MAHOUT-818
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-818.patch
>
>
> Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, -cf (--clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-818) Canopy Emits Too Many Trivial
Clusters
Posted by "beneo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125747#comment-13125747 ]
beneo commented on MAHOUT-818:
------------------------------
this path make the clustering result bad
i add the vector i used for the test, there is 5000 vectors
canopy t1 = 0.032 t2 = 0.008
kmeans x = 10 cd = 0.1
if clusterFilter = 0
the result is awesome
if clusterFilter = 1
OMG.....
However, the clusterFilter make the canopy very faster, it's important for the 3Billions vector canopy..
> Canopy Emits Too Many Trivial Clusters
> --------------------------------------
>
> Key: MAHOUT-818
> URL: https://issues.apache.org/jira/browse/MAHOUT-818
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-818.patch
>
>
> Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, -cf (--clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-818) Canopy Emits Too Many Trivial
Clusters
Posted by "beneo (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
beneo updated MAHOUT-818:
-------------------------
Attachment: vector.tar.gz
this is the vectors for my test.
you can use this to watch the different result with clusterFilter or not
the canopy t1=0.032, t2=0.008
the kmeans x=10 cd=0.1
> Canopy Emits Too Many Trivial Clusters
> --------------------------------------
>
> Key: MAHOUT-818
> URL: https://issues.apache.org/jira/browse/MAHOUT-818
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-818.patch, vector.tar.gz
>
>
> Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, -cf (--clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAHOUT-818) Canopy Emits Too Many Trivial
Clusters
Posted by "Jeff Eastman (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman resolved MAHOUT-818.
---------------------------------
Resolution: Fixed
Patch committed to trunk
> Canopy Emits Too Many Trivial Clusters
> --------------------------------------
>
> Key: MAHOUT-818
> URL: https://issues.apache.org/jira/browse/MAHOUT-818
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-818.patch
>
>
> Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, -cf (--clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-818) Canopy Emits Too Many Trivial
Clusters
Posted by "Jeff Eastman (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-818:
--------------------------------
Attachment: MAHOUT-818.patch
This patch implements a new Canopy CLI argument which limits the minimum size of output canopies. Unit tests have been updated and all run.
> Canopy Emits Too Many Trivial Clusters
> --------------------------------------
>
> Key: MAHOUT-818
> URL: https://issues.apache.org/jira/browse/MAHOUT-818
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-818.patch
>
>
> Users of Canopy clustering report that the single reducer used in the mapreduce version often takes dispropportately long to process the results of multiple mappers. This patch introduces a new Canopy CLI argument, -cf (--clusterFilter), which if present establishes a lower bound on the numPoints of canopies output from the algorithm. The default value for this filter is 0, and all canopies are output. Setting -cf 1 would eliminate any canopies which contain only 1 point from subsequent processing steps.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira