You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Shashikant Kore <sh...@gmail.com> on 2009/09/22 15:59:48 UTC
Understanding Canopy/Map Reduce
Hi,
I am unable to understand how the Canopy clustering works.
In Map stage, Canopy.addPointToCanopies() is called for every point
with list of canopies. This method adds to the existing canopy or
creates new one or both depending on the distance of the vector from
existing canopy centroids. Map stage outputs all the canopy centroids
(with key "centroid").
In reduce phase, these centroids will again undergo the same process
(so, possible merges) and finally centroids will be output'ed. But, I
see that in CanopyReducer the input values are the input vectors and
not the centroids received from the Map stage.
I think, I missing something here. Can you please let me know what it is?
Note: I am using CanopyDriver utility (and not CanopyClusteringJob).
Thanks,
--shashi
Re: Understanding Canopy/Map Reduce
Posted by Shashikant Kore <sh...@gmail.com>.
Thanks Grant, Jeff for clarification. I understand it now.
--shashi
On Tue, Sep 22, 2009 at 10:40 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Sep 22, 2009, at 9:59 AM, Shashikant Kore wrote:
>
>> Hi,
>>
>> I am unable to understand how the Canopy clustering works.
>>
>> In Map stage, Canopy.addPointToCanopies() is called for every point
>> with list of canopies. This method adds to the existing canopy or
>> creates new one or both depending on the distance of the vector from
>> existing canopy centroids. Map stage outputs all the canopy centroids
>> (with key "centroid").
>>
>> In reduce phase, these centroids will again undergo the same process
>> (so, possible merges) and finally centroids will be output'ed. But, I
>> see that in CanopyReducer the input values are the input vectors and
>> not the centroids received from the Map stage.
>
> If I recall correctly, the centroids get loaded up in the init stage of the
> Mapper and the Reducer, but I don't have the code open at the moment. Thus,
> the input vectors can then get associated with the centroids.
>
>>
>> I think, I missing something here. Can you please let me know what it is?
>>
>> Note: I am using CanopyDriver utility (and not CanopyClusteringJob).
>>
>> Thanks,
>>
>> --shashi
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
> Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
Re: Understanding Canopy/Map Reduce
Posted by Grant Ingersoll <gs...@apache.org>.
On Sep 22, 2009, at 9:59 AM, Shashikant Kore wrote:
> Hi,
>
> I am unable to understand how the Canopy clustering works.
>
> In Map stage, Canopy.addPointToCanopies() is called for every point
> with list of canopies. This method adds to the existing canopy or
> creates new one or both depending on the distance of the vector from
> existing canopy centroids. Map stage outputs all the canopy centroids
> (with key "centroid").
>
> In reduce phase, these centroids will again undergo the same process
> (so, possible merges) and finally centroids will be output'ed. But, I
> see that in CanopyReducer the input values are the input vectors and
> not the centroids received from the Map stage.
If I recall correctly, the centroids get loaded up in the init stage
of the Mapper and the Reducer, but I don't have the code open at the
moment. Thus, the input vectors can then get associated with the
centroids.
>
> I think, I missing something here. Can you please let me know what
> it is?
>
> Note: I am using CanopyDriver utility (and not CanopyClusteringJob).
>
> Thanks,
>
> --shashi
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: Understanding Canopy/Map Reduce
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Shashikant Kore wrote:
> Hi,
>
> I am unable to understand how the Canopy clustering works.
>
> In Map stage, Canopy.addPointToCanopies() is called for every point
> with list of canopies. This method adds to the existing canopy or
> creates new one or both depending on the distance of the vector from
> existing canopy centroids. Map stage outputs all the canopy centroids
> (with key "centroid").
>
> In reduce phase, these centroids will again undergo the same process
> (so, possible merges) and finally centroids will be output'ed. But, I
> see that in CanopyReducer the input values are the input vectors and
> not the centroids received from the Map stage.
>
> I think, I missing something here. Can you please let me know what it is?
>
You are correct in most of your analysis, but the vectors processed by
the reducer are the centroid vectors produced by the mapper(s). Since
multiple mappers may each see only a portion of the dataset, the reducer
does a final canopy cluster of the centroids so that similar centroids
get coalesced.
> Note: I am using CanopyDriver utility (and not CanopyClusteringJob).
>
> Thanks,
>
> --shashi
>
>
>