You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jeff Eastman (JIRA)" <ji...@apache.org> on 2011/06/29 18:49:28 UTC
[jira] [Created] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
MeanShift Cannot Use Multiple Reducers
--------------------------------------
Key: MAHOUT-749
URL: https://issues.apache.org/jira/browse/MAHOUT-749
Project: Mahout
Issue Type: Improvement
Reporter: Jeff Eastman
The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
RE: [jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by Jeff Eastman <je...@Narus.com>.
Canopy is a single-pass algorithm so producing multiple reducer output files would be an issue with that idea. With Canopy you need to adjust T1 and T2 (T3 and T4 too) so that you get a manageable number of clusters from each mapper. The reducer phase just ensures that mapper outputs which are close together are merged.
Mean Shift is pretty much an iterative canopy implementation, hence its name: MeanShiftCanopy. If the number of reducers is greater than the number of iterations (the patch does not enforce this) then it will produce a single cluster file when it completes. Other values would terminate with multiple output files that might contain merge-able overlaps.
-----Original Message------
From: Elmer Garduno (JIRA) [mailto:jira@apache.org]
Sent: Wednesday, June 29, 2011 12:30 PM
To: dev@mahout.apache.org
Subject: [jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057416#comment-13057416 ]
Elmer Garduno commented on MAHOUT-749:
--------------------------------------
Hi Jeff,
The CanopyDriver has the same problem, it also sets the numReducers=1 do you think that this kind of solution could also fix Canopy scalability issues?
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved MAHOUT-749.
------------------------------
Resolution: Fixed
Fix Version/s: 0.6
Jeff am I right this is done? I'm marking for 0.6 in any event since looks like you're well into this.
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Elmer Garduno (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057416#comment-13057416 ]
Elmer Garduno commented on MAHOUT-749:
--------------------------------------
Hi Jeff,
The CanopyDriver has the same problem, it also sets the numReducers=1 do you think that this kind of solution could also fix Canopy scalability issues?
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAHOUT-749 started by Jeff Eastman.
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-749:
--------------------------------
Attachment: MAHOUT-749.patch
This patch implements changes to the driver and mapper to utilize multiple reducers. The driver is modified to decrease the number of reducers in each iteration, finally to 1. The mapper is changed to send each of its outputs to a different reducer, depending upon the number deployed in the iteration. The unit tests are modified and run. This is ready for some experimentation with larger datasets and multiple reducers specified by -Dmapred.reduce.tasks.
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069336#comment-13069336 ]
Hudson commented on MAHOUT-749:
-------------------------------
Integrated in Mahout-Quality #953 (See [https://builds.apache.org/job/Mahout-Quality/953/])
MAHOUT-749: Implemented multiple reducer approach from Jira patch, plus a scalability enhancement to avoid accumulating merged clusterIds if -cl option is not present. The defaults are for the same behavior as before. All tests run though this needs more testing to see how it really scales
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1149369
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyMapper.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopy.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyClusterer.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/meanshift/TestMeanShift.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayMeanShift.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyDriver.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyConfigKeys.java
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-749:
-----------------------------
Component/s: Clustering
Affects Version/s: 0.5
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Components: Clustering
> Affects Versions: 0.5
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.6
>
> Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAHOUT-749) MeanShift Cannot Use Multiple
Reducers
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman reassigned MAHOUT-749:
-----------------------------------
Assignee: Jeff Eastman
> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
> Key: MAHOUT-749
> URL: https://issues.apache.org/jira/browse/MAHOUT-749
> Project: Mahout
> Issue Type: Improvement
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira