You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Grant Ingersoll <gs...@apache.org> on 2011/11/23 14:54:26 UTC
Re: MinHash Clustering in Mahout
On Oct 26, 2011, at 10:51 AM, Suneel Marthi wrote:
> I am still trying to fully understand minHash algorithm and I had the same results like below when running the MinHashDriver.
>
> I have a use case wherein I need to determine the content similarity of 2 documents like what's been described in Andrei Broder's paper - 'Identifying and Filtering Near-Duplicate Documents' (http://dl.acm.org/citation.cfm?id=736184).
>
> I started dissecting the clusters generated by Mahout's MinHashDriver to compare document content equality and to determine how accurate the clustering was?
> I do see that the first 2 files from the output below were put in the same cluster 106460162-207863047; thought the actual text content in both the files is different. How?
What do the vectors of these look like? If I remember correctly, some of those files don't have much actual text in them, such that I wonder if they are more or less empty. Running now to check.
>
> I am assuming that the NGram attribute was set to the default value of 1 when creating the tf-idf vectors from sequence files.
>
> Suneel
>
>
>
> ________________________________
> From: Grant Ingersoll <gs...@apache.org>
> To: user@mahout.apache.org
> Sent: Tuesday, October 25, 2011 5:55 AM
> Subject: Re: MinHash Clustering in Mahout
>
>
> On Oct 19, 2011, at 11:38 AM, Varun Thacker wrote:
>
>> I was trying to run the MinHash algorithm on the Reuters data set, so I did
>> the following before running MinHashDriver
>>
>> - Get the Reuters dataset
>> - Run org.apache.lucene.benchmark.utils.ExtractReuters to generate
>> reuters-out from reuters-sgm(the downloaded archive)
>> - Run seqdirectory to convert reuters-out to SequenceFile format
>> - Run seq2sparse to convert SequenceFiles to sparse vector format
>>
>> I used these instructions from the K-means clustering wiki page.
>>
>> This is the command I used to run MinHashDriver
>>
>> ./mahout org.apache.mahout.clustering.minhash.MinHashDriver --input
>> /home/varun/mahout/sparse/tfidf-vectors/ -o /home/varun/mahout/minhash
>>
>> The output file looks something like this:
>>
>> 106460162-207863047
> /reut2-015.sgm-653.txt
>> 106460162-207863047 /reut2-021.sgm-7.txt
>> 106460162-207863047 /reut2-013.sgm-307.txt
>> 106460162-207863047 /reut2-013.sgm-306.txt
>> 106460162-207863047 /reut2-014.sgm-786.txt
>> 106460162-207863047 /reut2-013.sgm-304.txt
>> 106460162-207863047 /reut2-013.sgm-303.txt
>> 106460162-207863047 /reut2-021.sgm-230.txt
>> 106460162-207863047 /reut2-012.sgm-548.txt
>> 106460162-207863047 /reut2-020.sgm-161.txt
>> 106460162-207863047 /reut2-021.sgm-553.txt
>> 106460162-207863047 /reut2-013.sgm-299.txt
>> 106460162-207863047 /reut2-015.sgm-284.txt
>> 106460162-207863047 /reut2-013.sgm-996.txt
>> 106460162-207863047 /reut2-021.sgm-441.txt
>> 106460162-207863047 /reut2-013.sgm-298.txt
>> 106460162-207863047 /reut2-013.sgm-995.txt
>> 106460162-207863047 /reut2-015.sgm-521.txt
>> 106460162-207863047 /reut2-020.sgm-162.txt
>> 106460162-207863047
> /reut2-020.sgm-163.txt
>> 106460162-207863047 /reut2-013.sgm-296.txt
>> ...
>> ...
>>
>>
>> Is this the correct way of running MinHash.
>>
>> If yes then I would update the wiki page
>> https://cwiki.apache.org/confluence/display/MAHOUT/Minhash+Clustering with
>> the instructions.
>>
>> Otherwise if someone could tell me on what am I doing wrong.
>
> I haven't looked into the code, but I get similar outputs, so I assume it is working. Might be good to incorporate this into the build-reuters.sh as well as try it on some other input.
>
> -Grant
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com