You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jeff Eastman (JIRA)" <ji...@apache.org> on 2011/06/29 18:49:28 UTC

[jira] [Created] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

MeanShift Cannot Use Multiple Reducers
--------------------------------------

                 Key: MAHOUT-749
                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
             Project: Mahout
          Issue Type: Improvement
            Reporter: Jeff Eastman


The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

RE: [jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by Jeff Eastman <je...@Narus.com>.
Canopy is a single-pass algorithm so producing multiple reducer output files would be an issue with that idea. With Canopy you need to adjust T1 and T2 (T3 and T4 too) so that you get a manageable number of clusters from each mapper. The reducer phase just ensures that mapper outputs which are close together are merged.

Mean Shift is pretty much an iterative canopy implementation, hence its name: MeanShiftCanopy. If the number of reducers is greater than the number of iterations (the patch does not enforce this) then it will produce a single cluster file when it completes. Other values would terminate with multiple output files that might contain merge-able overlaps.

-----Original Message------
From: Elmer Garduno (JIRA) [mailto:jira@apache.org] 
Sent: Wednesday, June 29, 2011 12:30 PM
To: dev@mahout.apache.org
Subject: [jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers


    [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057416#comment-13057416 ] 

Elmer Garduno commented on MAHOUT-749:
--------------------------------------

Hi Jeff, 

The CanopyDriver has the same problem, it also sets the numReducers=1 do you think that this kind of solution could also fix Canopy scalability issues?

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>         Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved MAHOUT-749.
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.6

Jeff am I right this is done? I'm marking for 0.6 in any event since looks like you're well into this.

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>             Fix For: 0.6
>
>         Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Elmer Garduno (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057416#comment-13057416 ] 

Elmer Garduno commented on MAHOUT-749:
--------------------------------------

Hi Jeff, 

The CanopyDriver has the same problem, it also sets the numReducers=1 do you think that this kind of solution could also fix Canopy scalability issues?

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>         Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Work started] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on MAHOUT-749 started by Jeff Eastman.

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Eastman updated MAHOUT-749:
--------------------------------

    Attachment: MAHOUT-749.patch

This patch implements changes to the driver and mapper to utilize multiple reducers. The driver is modified to decrease the number of reducers in each iteration, finally to 1. The mapper is changed to send each of its outputs to a different reducer, depending upon the number deployed in the iteration. The unit tests are modified and run. This is ready for some experimentation with larger datasets and multiple reducers specified by -Dmapred.reduce.tasks.

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>         Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069336#comment-13069336 ] 

Hudson commented on MAHOUT-749:
-------------------------------

Integrated in Mahout-Quality #953 (See [https://builds.apache.org/job/Mahout-Quality/953/])
    MAHOUT-749: Implemented multiple reducer approach from Jira patch, plus a scalability enhancement to avoid accumulating merged clusterIds if -cl option is not present. The defaults are for the same behavior as before.  All tests run though this needs more testing to see how it really scales

jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1149369
Files : 
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyMapper.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopy.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyClusterer.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/clustering/meanshift/TestMeanShift.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/display/DisplayMeanShift.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyDriver.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/meanshift/MeanShiftCanopyConfigKeys.java


> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>         Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated MAHOUT-749:
-----------------------------

          Component/s: Clustering
    Affects Version/s: 0.5

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Clustering
>    Affects Versions: 0.5
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>             Fix For: 0.6
>
>         Attachments: MAHOUT-749.patch
>
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Assigned] (MAHOUT-749) MeanShift Cannot Use Multiple Reducers

Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAHOUT-749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Eastman reassigned MAHOUT-749:
-----------------------------------

    Assignee: Jeff Eastman

> MeanShift Cannot Use Multiple Reducers
> --------------------------------------
>
>                 Key: MAHOUT-749
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-749
>             Project: Mahout
>          Issue Type: Improvement
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>
> The MeanShiftCanopy clustering job sets the numReducers=1 and this severely limits its scalability for larger jobs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira