You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Michael Kazekin <Mi...@mediainsight.info> on 2012/05/24 14:29:46 UTC
Continue K-Means on pseudo-distributed mode after a server failure
Hi!
I have run a K-Means on a single node (pseudo-distributed mode) and
experienced a server failure.
Is there any way to continue clustering with existing M/R jobs?
Cheers,
Michael.
Re: Clustering a large crawl
Posted by Ted Dunning <te...@gmail.com>.
The information lost is another way of saying that the metric is
approximate.
The loss may be good if it is likely to improve generalization to new data
(and if new data are important to you) or may be bad if we are throwing
away real structure.
On Tue, Jun 5, 2012 at 6:35 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> As one paper puts it "For many datasets, the first several PCs explain
> most of the variance, so that the rest can be disregarded with minimal loss
> of information." Often much of the information lost is equivalent to noise,
> no?
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
I suspect my clustering problem is not the metric but noise in the data.
For instance Canopy cannot find reasonable clusters in the data. It
always produces 1-3 docs per cluster no matter the choice of t. This
leads me to imagine the data as an all too uniform cloud where the noise
is potentially obscuring the signal. Yes, I assume there is a signal in
there because random choices of k using kmeans produces some good
clusters (some bad too).
You say that dimensional reduction is primarily a tool to speed up
certain analysis. While it certainly does that, I was using the term to
also include techniques like principal component analysis using SVD. As
one paper puts it "For many datasets, the first several PCs explain most
of the variance, so that the rest can be disregarded with minimal loss
of information." Often much of the information lost is equivalent to
noise, no?
On 6/5/12 12:59 AM, Ted Dunning wrote:
> Yes. This indicates that you should be able to get good clustering from
> that metric.
>
> On Mon, Jun 4, 2012 at 10:58 PM, Pat Ferrel<pa...@occamsmachete.com> wrote:
>
>> I do have rowsimilarity calculated for each doc using the same measure as
>> clustering and that produces pretty good results as far as the eyeball take
>> you. I assume this is what you mean by using the doc as a query.
>>
>> On 6/4/12 9:14 AM, Ted Dunning wrote:
>>
>>> Even having millions of dimensions isn't all that bad if that induces a
>>> reasonable distance between documents. The easy way to test that is to
>>> use
>>> several document vectors as queries and see whether the closest other
>>> documents appear to you to be very similar. If this is true for a number
>>> of documents, you should be good to go with whatever metric you are using.
>>>
>>> For fast clustering, you may need a low-dimensional surrogate metric so
>>> that you can get higher throughput, but the point of the low-dimensional
>>> surrogate is that it *replicates* the behavior of the metric that you
>>> really want. It isn't going to make your metric better.
>>>
>>> On Mon, Jun 4, 2012 at 5:15 PM, Pat Ferrel<pa...@occamsmachete.com> wrote:
>>>
>>> After looking again at the dictionary for 150,000 web pages I have
>>>> 259,000
>>>> dimensions! Part of the problem is I can't get Tika to detect language
>>>> very
>>>> well (working on this) so I get groups of non-english pages that throw in
>>>> quite a few new terms. Overall I think some form of dimensional reduction
>>>> is called for, no?
>>>>
Re: Clustering a large crawl
Posted by Ted Dunning <te...@gmail.com>.
Yes. This indicates that you should be able to get good clustering from
that metric.
On Mon, Jun 4, 2012 at 10:58 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> I do have rowsimilarity calculated for each doc using the same measure as
> clustering and that produces pretty good results as far as the eyeball take
> you. I assume this is what you mean by using the doc as a query.
>
> On 6/4/12 9:14 AM, Ted Dunning wrote:
>
>> Even having millions of dimensions isn't all that bad if that induces a
>> reasonable distance between documents. The easy way to test that is to
>> use
>> several document vectors as queries and see whether the closest other
>> documents appear to you to be very similar. If this is true for a number
>> of documents, you should be good to go with whatever metric you are using.
>>
>> For fast clustering, you may need a low-dimensional surrogate metric so
>> that you can get higher throughput, but the point of the low-dimensional
>> surrogate is that it *replicates* the behavior of the metric that you
>> really want. It isn't going to make your metric better.
>>
>> On Mon, Jun 4, 2012 at 5:15 PM, Pat Ferrel<pa...@occamsmachete.com> wrote:
>>
>> After looking again at the dictionary for 150,000 web pages I have
>>> 259,000
>>> dimensions! Part of the problem is I can't get Tika to detect language
>>> very
>>> well (working on this) so I get groups of non-english pages that throw in
>>> quite a few new terms. Overall I think some form of dimensional reduction
>>> is called for, no?
>>>
>>
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
I do have rowsimilarity calculated for each doc using the same measure
as clustering and that produces pretty good results as far as the
eyeball take you. I assume this is what you mean by using the doc as a
query.
On 6/4/12 9:14 AM, Ted Dunning wrote:
> Even having millions of dimensions isn't all that bad if that induces a
> reasonable distance between documents. The easy way to test that is to use
> several document vectors as queries and see whether the closest other
> documents appear to you to be very similar. If this is true for a number
> of documents, you should be good to go with whatever metric you are using.
>
> For fast clustering, you may need a low-dimensional surrogate metric so
> that you can get higher throughput, but the point of the low-dimensional
> surrogate is that it *replicates* the behavior of the metric that you
> really want. It isn't going to make your metric better.
>
> On Mon, Jun 4, 2012 at 5:15 PM, Pat Ferrel<pa...@occamsmachete.com> wrote:
>
>> After looking again at the dictionary for 150,000 web pages I have 259,000
>> dimensions! Part of the problem is I can't get Tika to detect language very
>> well (working on this) so I get groups of non-english pages that throw in
>> quite a few new terms. Overall I think some form of dimensional reduction
>> is called for, no?
Re: Clustering a large crawl
Posted by Ted Dunning <te...@gmail.com>.
Even having millions of dimensions isn't all that bad if that induces a
reasonable distance between documents. The easy way to test that is to use
several document vectors as queries and see whether the closest other
documents appear to you to be very similar. If this is true for a number
of documents, you should be good to go with whatever metric you are using.
For fast clustering, you may need a low-dimensional surrogate metric so
that you can get higher throughput, but the point of the low-dimensional
surrogate is that it *replicates* the behavior of the metric that you
really want. It isn't going to make your metric better.
On Mon, Jun 4, 2012 at 5:15 PM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> After looking again at the dictionary for 150,000 web pages I have 259,000
> dimensions! Part of the problem is I can't get Tika to detect language very
> well (working on this) so I get groups of non-english pages that throw in
> quite a few new terms. Overall I think some form of dimensional reduction
> is called for, no?
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
True and lucene stop words are small in number. Lucene I assume wants to
be able to search on words that are only important in the search phrase,
not to TFIDF so they have only a small number. I got a big list of stop
words from another source. But as you say it only helps a little.
In the same vein as Hapax_legomenon...
On a previous project we used a seperate IDF calculation for each
domain. The domain specific IDF weighted frequent terms specific to the
site lower, reducing the impact of meanless catch-phrases or site
specific jargon that had did not apply globally. There does not seem to
be an easy way to do this with mahout. There were some issues with doing
domain-specific IDF anyway, like throwing out important words when they
weren't meaningless catch phrases.
Lance, Yes, I understand that n-grams give more terms, bad phrasing on
my part. I have a very high mll though so it shouldn't increase the
terms by very much.
After looking again at the dictionary for 150,000 web pages I have
259,000 dimensions! Part of the problem is I can't get Tika to detect
language very well (working on this) so I get groups of non-english
pages that throw in quite a few new terms. Overall I think some form of
dimensional reduction is called for, no?
On 6/3/12 12:34 AM, Ted Dunning wrote:
> Also, stop words usually only makes it minutely smaller.
>
> What really makes a vocabulary smaller is eliminating
> hapax<http://en.wikipedia.org/wiki/Hapax_legomenon>
> .
>
> On Sun, Jun 3, 2012 at 9:23 AM, Lance Norskog<go...@gmail.com> wrote:
>
>> "100,000 terms even with n-grams, "...
>>
>> Ummmm... N-grams will make it bigger, not smaller :)
>>
>> I haven't studied the text workflows lately. Is there a place where you get
>> counts for all words? If so, you can just pick the smallest N counts and
>> make a stopword list out of them. This would be a highly valued addition to
>> the workflows.
>>
>> On Fri, Jun 1, 2012 at 9:36 AM, Ted Dunning<te...@gmail.com> wrote:
>>
>>> I am pretty sure that Suneel meant keep the top 1000 terms per document.
>>>
>>>
>>> On Fri, Jun 1, 2012 at 2:21 AM, Pat Ferrel<pa...@occamsmachete.com>
>> wrote:
>>>> Are you saying that
>>>> 1. you threw out all but the top 1000 terms per document by weight? or
>>>> 2. your dictionary has only 1000 terms in it and you threw all others
>>>> away?
>>>>
>>>> The later is a simple dimensional reduction trick to try, but 1000 seems
>>>> low to me for the entire dictionary.
>>>>
>>>> A question for you about similarity. I wonder if using all terms is
>>>> better for the similarity measure? What is noise in clustering may be
>>>> important when looking at cooccurrences. What do you think?
>>>>
>>>>
>>>> On 5/31/12 4:20 PM, Suneel Marthi wrote:
>>>>
>>>> Pat,
>>>>
>>>> We have been trying to do something very similar to what u r trying to
>>>> accomplish and we ended up with better clusters by considering only the
>> top
>>>> 1000 terms (by tf-idf weight) per doc and using Tanimoto distance.
>>>>
>>>> Definitely give dimensionality reduction a try and let us know how it
>>>> works out.
>>>>
>>>> ------------------------------
>>>> *From:* Pat Ferrel<pa...@occamsmachete.com> <pa...@occamsmachete.com>
>>>> *To:* user@mahout.apache.org
>>>> *Sent:* Thursday, May 31, 2012 6:42 PM
>>>> *Subject:* Re: Clustering a large crawl
>>>>
>>>> Yeah, that's the conclusion I was coming to but thought I'd ask the
>>>> experts. My dictionary is petty big. the last time I looked it was over
>>>> 100,000 terms even with n-grams, lucene stop words, no numbers, and
>>>> stemming. I've tried Tanimoto too with similar results.
>>>>
>>>> Dimensional reduction seems like the next thing to try.
>>>>
>>>> -Pat
>>>>
>>>>
>>>> Further data from 150,000 docs. Using Canopy clustering I get these
>> values
>>>> t1 = t2 = 0.3 => 123094 canopies
>>>> t1 = t2 = 0.6 => 97035 canopies
>>>> t1 = t2 = 0.9 => 60160 canopies
>>>> t1 = t2 = 0.91 => 59491 canopies
>>>> t1 = t2 = 0.93 => 58526 canopies
>>>> t1 = t2 = 0.95 => 57854 canopies
>>>> t1 = t2 = 0.97 => 57244 canopies
>>>> t1 = t2 = 0.99 => 56241 canopies
>>>>
>>>>
>>>>
>>>> On 5/31/12 2:31 PM, Jeff Eastman wrote:
>>>>
>>>> And I misconstrued your earlier remarks on cluster size vs number of
>>>> clusters. As t -> 1 you will get fewer and fewer canopies as you have
>>>> observed. It actually doesn't seem like the cosine distance measure is
>>>> working very well for you.
>>>>
>>>> Have you mentioned the size of your dictionary earlier? Perhaps
>>>> increasing the number of stop words that are rejected will decrease the
>>>> vector size and make clustering work better. This seems like the curse
>> of
>>>> dimensionality at work.
>>>>
>>>> On 5/31/12 11:18 AM, Pat Ferrel wrote:
>>>>
>>>> Oops, misspoke. 0 good, 1 bad for clustering at least
>>>> For similarity 1 good 0 bad.
>>>>
>>>> One is a similarity value and the other a distance measure.
>>>>
>>>> But the primary question is how to get better canopies. I would expect
>>>> that as the distance t gets small the number of canopies gets large
>> which
>>>> is what I see in the data below. Jeff suggests I try much smaller t to
>> get
>>>> less canopies and I will though I don't understand the logic. The docs
>> are
>>>> not all that similar. being from a general news crawl.
>>>>
>>>> When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
>>>> docs I get:
>>>> t1 = t2 = 0.3 => 123094 canopies
>>>> t1 = t2 = 0.6 => 97035 canopies
>>>> t1 = t2 = 0.9 => 60160 canopies
>>>>
>>>> Obviously none of these values for t is very useful and it looks like I
>>>> need to make t even larger, which would seem to indicate very
>>>> loose/non-dense canopies, no? For very large ts are the canopies useful?
>>>>
>>>> I'm trying both but the other odd thing is that it takes longer to run
>>>> canopy on this data than to run kmeans, a lot longer.
>>>>
>>>> On 5/31/12 12:44 AM, Sean Owen wrote:
>>>>
>>>> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com><
>> pat@occamsmachete.com>
>>>> wrote:
>>>>
>>>> I see
>>>> double denominator = Math.sqrt(lengthSquaredp1) *
>>>> Math.sqrt(lengthSquaredp2);
>>>> // correct for floating-point rounding errors
>>>> if (denominator< dotProduct) {
>>>> denominator = dotProduct;
>>>> }
>>>> return 1.0 - dotProduct / denominator;
>>>>
>>>> So this is going to return 1 - cosine, right? So for clustering the
>>>> distance 1 = very close, 0 = very far.
>>>>
>>>>
>>>> When two vectors are close, the angle between them is small, so the
>>>> cosine
>>>> is large, near 1. 0 = close, 1 = far, as expected.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
Re: Clustering a large crawl
Posted by Ted Dunning <te...@gmail.com>.
Also, stop words usually only makes it minutely smaller.
What really makes a vocabulary smaller is eliminating
hapax<http://en.wikipedia.org/wiki/Hapax_legomenon>
.
On Sun, Jun 3, 2012 at 9:23 AM, Lance Norskog <go...@gmail.com> wrote:
> "100,000 terms even with n-grams, "...
>
> Ummmm... N-grams will make it bigger, not smaller :)
>
> I haven't studied the text workflows lately. Is there a place where you get
> counts for all words? If so, you can just pick the smallest N counts and
> make a stopword list out of them. This would be a highly valued addition to
> the workflows.
>
> On Fri, Jun 1, 2012 at 9:36 AM, Ted Dunning <te...@gmail.com> wrote:
>
> > I am pretty sure that Suneel meant keep the top 1000 terms per document.
> >
> >
> > On Fri, Jun 1, 2012 at 2:21 AM, Pat Ferrel <pa...@occamsmachete.com>
> wrote:
> >
> >> Are you saying that
> >> 1. you threw out all but the top 1000 terms per document by weight? or
> >> 2. your dictionary has only 1000 terms in it and you threw all others
> >> away?
> >>
> >> The later is a simple dimensional reduction trick to try, but 1000 seems
> >> low to me for the entire dictionary.
> >>
> >> A question for you about similarity. I wonder if using all terms is
> >> better for the similarity measure? What is noise in clustering may be
> >> important when looking at cooccurrences. What do you think?
> >>
> >>
> >> On 5/31/12 4:20 PM, Suneel Marthi wrote:
> >>
> >> Pat,
> >>
> >> We have been trying to do something very similar to what u r trying to
> >> accomplish and we ended up with better clusters by considering only the
> top
> >> 1000 terms (by tf-idf weight) per doc and using Tanimoto distance.
> >>
> >> Definitely give dimensionality reduction a try and let us know how it
> >> works out.
> >>
> >> ------------------------------
> >> *From:* Pat Ferrel <pa...@occamsmachete.com> <pa...@occamsmachete.com>
> >> *To:* user@mahout.apache.org
> >> *Sent:* Thursday, May 31, 2012 6:42 PM
> >> *Subject:* Re: Clustering a large crawl
> >>
> >> Yeah, that's the conclusion I was coming to but thought I'd ask the
> >> experts. My dictionary is petty big. the last time I looked it was over
> >> 100,000 terms even with n-grams, lucene stop words, no numbers, and
> >> stemming. I've tried Tanimoto too with similar results.
> >>
> >> Dimensional reduction seems like the next thing to try.
> >>
> >> -Pat
> >>
> >>
> >> Further data from 150,000 docs. Using Canopy clustering I get these
> values
> >> t1 = t2 = 0.3 => 123094 canopies
> >> t1 = t2 = 0.6 => 97035 canopies
> >> t1 = t2 = 0.9 => 60160 canopies
> >> t1 = t2 = 0.91 => 59491 canopies
> >> t1 = t2 = 0.93 => 58526 canopies
> >> t1 = t2 = 0.95 => 57854 canopies
> >> t1 = t2 = 0.97 => 57244 canopies
> >> t1 = t2 = 0.99 => 56241 canopies
> >>
> >>
> >>
> >> On 5/31/12 2:31 PM, Jeff Eastman wrote:
> >>
> >> And I misconstrued your earlier remarks on cluster size vs number of
> >> clusters. As t -> 1 you will get fewer and fewer canopies as you have
> >> observed. It actually doesn't seem like the cosine distance measure is
> >> working very well for you.
> >>
> >> Have you mentioned the size of your dictionary earlier? Perhaps
> >> increasing the number of stop words that are rejected will decrease the
> >> vector size and make clustering work better. This seems like the curse
> of
> >> dimensionality at work.
> >>
> >> On 5/31/12 11:18 AM, Pat Ferrel wrote:
> >>
> >> Oops, misspoke. 0 good, 1 bad for clustering at least
> >> For similarity 1 good 0 bad.
> >>
> >> One is a similarity value and the other a distance measure.
> >>
> >> But the primary question is how to get better canopies. I would expect
> >> that as the distance t gets small the number of canopies gets large
> which
> >> is what I see in the data below. Jeff suggests I try much smaller t to
> get
> >> less canopies and I will though I don't understand the logic. The docs
> are
> >> not all that similar. being from a general news crawl.
> >>
> >> When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
> >> docs I get:
> >> t1 = t2 = 0.3 => 123094 canopies
> >> t1 = t2 = 0.6 => 97035 canopies
> >> t1 = t2 = 0.9 => 60160 canopies
> >>
> >> Obviously none of these values for t is very useful and it looks like I
> >> need to make t even larger, which would seem to indicate very
> >> loose/non-dense canopies, no? For very large ts are the canopies useful?
> >>
> >> I'm trying both but the other odd thing is that it takes longer to run
> >> canopy on this data than to run kmeans, a lot longer.
> >>
> >> On 5/31/12 12:44 AM, Sean Owen wrote:
> >>
> >> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com><
> pat@occamsmachete.com>
> >> wrote:
> >>
> >> I see
> >> double denominator = Math.sqrt(lengthSquaredp1) *
> >> Math.sqrt(lengthSquaredp2);
> >> // correct for floating-point rounding errors
> >> if (denominator< dotProduct) {
> >> denominator = dotProduct;
> >> }
> >> return 1.0 - dotProduct / denominator;
> >>
> >> So this is going to return 1 - cosine, right? So for clustering the
> >> distance 1 = very close, 0 = very far.
> >>
> >>
> >> When two vectors are close, the angle between them is small, so the
> >> cosine
> >> is large, near 1. 0 = close, 1 = far, as expected.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Clustering a large crawl
Posted by Lance Norskog <go...@gmail.com>.
"100,000 terms even with n-grams, "...
Ummmm... N-grams will make it bigger, not smaller :)
I haven't studied the text workflows lately. Is there a place where you get
counts for all words? If so, you can just pick the smallest N counts and
make a stopword list out of them. This would be a highly valued addition to
the workflows.
On Fri, Jun 1, 2012 at 9:36 AM, Ted Dunning <te...@gmail.com> wrote:
> I am pretty sure that Suneel meant keep the top 1000 terms per document.
>
>
> On Fri, Jun 1, 2012 at 2:21 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
>> Are you saying that
>> 1. you threw out all but the top 1000 terms per document by weight? or
>> 2. your dictionary has only 1000 terms in it and you threw all others
>> away?
>>
>> The later is a simple dimensional reduction trick to try, but 1000 seems
>> low to me for the entire dictionary.
>>
>> A question for you about similarity. I wonder if using all terms is
>> better for the similarity measure? What is noise in clustering may be
>> important when looking at cooccurrences. What do you think?
>>
>>
>> On 5/31/12 4:20 PM, Suneel Marthi wrote:
>>
>> Pat,
>>
>> We have been trying to do something very similar to what u r trying to
>> accomplish and we ended up with better clusters by considering only the top
>> 1000 terms (by tf-idf weight) per doc and using Tanimoto distance.
>>
>> Definitely give dimensionality reduction a try and let us know how it
>> works out.
>>
>> ------------------------------
>> *From:* Pat Ferrel <pa...@occamsmachete.com> <pa...@occamsmachete.com>
>> *To:* user@mahout.apache.org
>> *Sent:* Thursday, May 31, 2012 6:42 PM
>> *Subject:* Re: Clustering a large crawl
>>
>> Yeah, that's the conclusion I was coming to but thought I'd ask the
>> experts. My dictionary is petty big. the last time I looked it was over
>> 100,000 terms even with n-grams, lucene stop words, no numbers, and
>> stemming. I've tried Tanimoto too with similar results.
>>
>> Dimensional reduction seems like the next thing to try.
>>
>> -Pat
>>
>>
>> Further data from 150,000 docs. Using Canopy clustering I get these values
>> t1 = t2 = 0.3 => 123094 canopies
>> t1 = t2 = 0.6 => 97035 canopies
>> t1 = t2 = 0.9 => 60160 canopies
>> t1 = t2 = 0.91 => 59491 canopies
>> t1 = t2 = 0.93 => 58526 canopies
>> t1 = t2 = 0.95 => 57854 canopies
>> t1 = t2 = 0.97 => 57244 canopies
>> t1 = t2 = 0.99 => 56241 canopies
>>
>>
>>
>> On 5/31/12 2:31 PM, Jeff Eastman wrote:
>>
>> And I misconstrued your earlier remarks on cluster size vs number of
>> clusters. As t -> 1 you will get fewer and fewer canopies as you have
>> observed. It actually doesn't seem like the cosine distance measure is
>> working very well for you.
>>
>> Have you mentioned the size of your dictionary earlier? Perhaps
>> increasing the number of stop words that are rejected will decrease the
>> vector size and make clustering work better. This seems like the curse of
>> dimensionality at work.
>>
>> On 5/31/12 11:18 AM, Pat Ferrel wrote:
>>
>> Oops, misspoke. 0 good, 1 bad for clustering at least
>> For similarity 1 good 0 bad.
>>
>> One is a similarity value and the other a distance measure.
>>
>> But the primary question is how to get better canopies. I would expect
>> that as the distance t gets small the number of canopies gets large which
>> is what I see in the data below. Jeff suggests I try much smaller t to get
>> less canopies and I will though I don't understand the logic. The docs are
>> not all that similar. being from a general news crawl.
>>
>> When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
>> docs I get:
>> t1 = t2 = 0.3 => 123094 canopies
>> t1 = t2 = 0.6 => 97035 canopies
>> t1 = t2 = 0.9 => 60160 canopies
>>
>> Obviously none of these values for t is very useful and it looks like I
>> need to make t even larger, which would seem to indicate very
>> loose/non-dense canopies, no? For very large ts are the canopies useful?
>>
>> I'm trying both but the other odd thing is that it takes longer to run
>> canopy on this data than to run kmeans, a lot longer.
>>
>> On 5/31/12 12:44 AM, Sean Owen wrote:
>>
>> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com>
>> wrote:
>>
>> I see
>> double denominator = Math.sqrt(lengthSquaredp1) *
>> Math.sqrt(lengthSquaredp2);
>> // correct for floating-point rounding errors
>> if (denominator< dotProduct) {
>> denominator = dotProduct;
>> }
>> return 1.0 - dotProduct / denominator;
>>
>> So this is going to return 1 - cosine, right? So for clustering the
>> distance 1 = very close, 0 = very far.
>>
>>
>> When two vectors are close, the angle between them is small, so the
>> cosine
>> is large, near 1. 0 = close, 1 = far, as expected.
>>
>>
>>
>>
>>
>>
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Clustering a large crawl
Posted by Ted Dunning <te...@gmail.com>.
I am pretty sure that Suneel meant keep the top 1000 terms per document.
On Fri, Jun 1, 2012 at 2:21 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> Are you saying that
> 1. you threw out all but the top 1000 terms per document by weight? or
> 2. your dictionary has only 1000 terms in it and you threw all others
> away?
>
> The later is a simple dimensional reduction trick to try, but 1000 seems
> low to me for the entire dictionary.
>
> A question for you about similarity. I wonder if using all terms is better
> for the similarity measure? What is noise in clustering may be important
> when looking at cooccurrences. What do you think?
>
>
> On 5/31/12 4:20 PM, Suneel Marthi wrote:
>
> Pat,
>
> We have been trying to do something very similar to what u r trying to
> accomplish and we ended up with better clusters by considering only the top
> 1000 terms (by tf-idf weight) per doc and using Tanimoto distance.
>
> Definitely give dimensionality reduction a try and let us know how it
> works out.
>
> ------------------------------
> *From:* Pat Ferrel <pa...@occamsmachete.com> <pa...@occamsmachete.com>
> *To:* user@mahout.apache.org
> *Sent:* Thursday, May 31, 2012 6:42 PM
> *Subject:* Re: Clustering a large crawl
>
> Yeah, that's the conclusion I was coming to but thought I'd ask the
> experts. My dictionary is petty big. the last time I looked it was over
> 100,000 terms even with n-grams, lucene stop words, no numbers, and
> stemming. I've tried Tanimoto too with similar results.
>
> Dimensional reduction seems like the next thing to try.
>
> -Pat
>
>
> Further data from 150,000 docs. Using Canopy clustering I get these values
> t1 = t2 = 0.3 => 123094 canopies
> t1 = t2 = 0.6 => 97035 canopies
> t1 = t2 = 0.9 => 60160 canopies
> t1 = t2 = 0.91 => 59491 canopies
> t1 = t2 = 0.93 => 58526 canopies
> t1 = t2 = 0.95 => 57854 canopies
> t1 = t2 = 0.97 => 57244 canopies
> t1 = t2 = 0.99 => 56241 canopies
>
>
>
> On 5/31/12 2:31 PM, Jeff Eastman wrote:
>
> And I misconstrued your earlier remarks on cluster size vs number of
> clusters. As t -> 1 you will get fewer and fewer canopies as you have
> observed. It actually doesn't seem like the cosine distance measure is
> working very well for you.
>
> Have you mentioned the size of your dictionary earlier? Perhaps increasing
> the number of stop words that are rejected will decrease the vector size
> and make clustering work better. This seems like the curse of
> dimensionality at work.
>
> On 5/31/12 11:18 AM, Pat Ferrel wrote:
>
> Oops, misspoke. 0 good, 1 bad for clustering at least
> For similarity 1 good 0 bad.
>
> One is a similarity value and the other a distance measure.
>
> But the primary question is how to get better canopies. I would expect
> that as the distance t gets small the number of canopies gets large which
> is what I see in the data below. Jeff suggests I try much smaller t to get
> less canopies and I will though I don't understand the logic. The docs are
> not all that similar. being from a general news crawl.
>
> When using the CosineDistanceMeasure in Canopy on a corpus of 150,000 docs
> I get:
> t1 = t2 = 0.3 => 123094 canopies
> t1 = t2 = 0.6 => 97035 canopies
> t1 = t2 = 0.9 => 60160 canopies
>
> Obviously none of these values for t is very useful and it looks like I
> need to make t even larger, which would seem to indicate very
> loose/non-dense canopies, no? For very large ts are the canopies useful?
>
> I'm trying both but the other odd thing is that it takes longer to run
> canopy on this data than to run kmeans, a lot longer.
>
> On 5/31/12 12:44 AM, Sean Owen wrote:
>
> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com>
> wrote:
>
> I see
> double denominator = Math.sqrt(lengthSquaredp1) *
> Math.sqrt(lengthSquaredp2);
> // correct for floating-point rounding errors
> if (denominator< dotProduct) {
> denominator = dotProduct;
> }
> return 1.0 - dotProduct / denominator;
>
> So this is going to return 1 - cosine, right? So for clustering the
> distance 1 = very close, 0 = very far.
>
>
> When two vectors are close, the angle between them is small, so the
> cosine
> is large, near 1. 0 = close, 1 = far, as expected.
>
>
>
>
>
>
>
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Are you saying that
1. you threw out all but the top 1000 terms per document by weight? or
2. your dictionary has only 1000 terms in it and you threw all others
away?
The later is a simple dimensional reduction trick to try, but 1000 seems
low to me for the entire dictionary.
A question for you about similarity. I wonder if using all terms is
better for the similarity measure? What is noise in clustering may be
important when looking at cooccurrences. What do you think?
On 5/31/12 4:20 PM, Suneel Marthi wrote:
> Pat,
>
> We have been trying to do something very similar to what u r trying to
> accomplish and we ended up with better clusters by considering only
> the top 1000 terms (by tf-idf weight) per doc and using Tanimoto
> distance.
>
> Definitely give dimensionality reduction a try and let us know how it
> works out.
>
> ------------------------------------------------------------------------
> *From:* Pat Ferrel <pa...@occamsmachete.com>
> *To:* user@mahout.apache.org
> *Sent:* Thursday, May 31, 2012 6:42 PM
> *Subject:* Re: Clustering a large crawl
>
> Yeah, that's the conclusion I was coming to but thought I'd ask the
> experts. My dictionary is petty big. the last time I looked it was
> over 100,000 terms even with n-grams, lucene stop words, no numbers,
> and stemming. I've tried Tanimoto too with similar results.
>
> Dimensional reduction seems like the next thing to try.
>
> -Pat
>
>
> Further data from 150,000 docs. Using Canopy clustering I get these values
> t1 = t2 = 0.3 => 123094 canopies
> t1 = t2 = 0.6 => 97035 canopies
> t1 = t2 = 0.9 => 60160 canopies
> t1 = t2 = 0.91 => 59491 canopies
> t1 = t2 = 0.93 => 58526 canopies
> t1 = t2 = 0.95 => 57854 canopies
> t1 = t2 = 0.97 => 57244 canopies
> t1 = t2 = 0.99 => 56241 canopies
>
>
>
> On 5/31/12 2:31 PM, Jeff Eastman wrote:
>> And I misconstrued your earlier remarks on cluster size vs number of
>> clusters. As t -> 1 you will get fewer and fewer canopies as you have
>> observed. It actually doesn't seem like the cosine distance measure
>> is working very well for you.
>>
>> Have you mentioned the size of your dictionary earlier? Perhaps
>> increasing the number of stop words that are rejected will decrease
>> the vector size and make clustering work better. This seems like the
>> curse of dimensionality at work.
>>
>> On 5/31/12 11:18 AM, Pat Ferrel wrote:
>>> Oops, misspoke. 0 good, 1 bad for clustering at least
>>> For similarity 1 good 0 bad.
>>>
>>> One is a similarity value and the other a distance measure.
>>>
>>> But the primary question is how to get better canopies. I would
>>> expect that as the distance t gets small the number of canopies gets
>>> large which is what I see in the data below. Jeff suggests I try
>>> much smaller t to get less canopies and I will though I don't
>>> understand the logic. The docs are not all that similar. being from
>>> a general news crawl.
>>>
>>> When using the CosineDistanceMeasure in Canopy on a corpus of
>>> 150,000 docs I get:
>>> t1 = t2 = 0.3 => 123094 canopies
>>> t1 = t2 = 0.6 => 97035 canopies
>>> t1 = t2 = 0.9 => 60160 canopies
>>>
>>> Obviously none of these values for t is very useful and it looks
>>> like I need to make t even larger, which would seem to indicate very
>>> loose/non-dense canopies, no? For very large ts are the canopies
>>> useful?
>>>
>>> I'm trying both but the other odd thing is that it takes longer to
>>> run canopy on this data than to run kmeans, a lot longer.
>>>
>>> On 5/31/12 12:44 AM, Sean Owen wrote:
>>>> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com>
>>>> <ma...@occamsmachete.com> wrote:
>>>>
>>>>> I see
>>>>> double denominator = Math.sqrt(lengthSquaredp1) *
>>>>> Math.sqrt(lengthSquaredp2);
>>>>> // correct for floating-point rounding errors
>>>>> if (denominator< dotProduct) {
>>>>> denominator = dotProduct;
>>>>> }
>>>>> return 1.0 - dotProduct / denominator;
>>>>>
>>>>> So this is going to return 1 - cosine, right? So for clustering the
>>>>> distance 1 = very close, 0 = very far.
>>>>>
>>>>>
>>>> When two vectors are close, the angle between them is small, so the
>>>> cosine
>>>> is large, near 1. 0 = close, 1 = far, as expected.
>>>>
>>>
>>>
>>
>
>
Re: Clustering a large crawl
Posted by Suneel Marthi <su...@yahoo.com>.
Pat,
We have been trying to do something very similar to what u r trying to accomplish and we ended up with better clusters by considering only the top 1000 terms (by tf-idf weight) per doc and using Tanimoto distance.
Definitely give dimensionality reduction a try and let us know how it works out.
________________________________
From: Pat Ferrel <pa...@occamsmachete.com>
To: user@mahout.apache.org
Sent: Thursday, May 31, 2012 6:42 PM
Subject: Re: Clustering a large crawl
Yeah, that's the conclusion I was coming to but thought I'd ask the experts. My dictionary is petty big. the last time I looked it was over 100,000 terms even with n-grams, lucene stop words, no numbers, and stemming. I've tried Tanimoto too with similar results.
Dimensional reduction seems like the next thing to try.
-Pat
Further data from 150,000 docs. Using Canopy clustering I get these
values
t1 = t2 = 0.3 => 123094 canopies
t1 = t2 = 0.6 => 97035 canopies
t1 = t2 = 0.9 => 60160 canopies
t1 = t2 = 0.91 => 59491 canopies
t1 = t2 = 0.93 => 58526 canopies
t1 = t2 = 0.95 => 57854 canopies
t1 = t2 = 0.97 => 57244 canopies
t1 = t2 = 0.99 => 56241 canopies
On 5/31/12 2:31 PM, Jeff Eastman wrote:
And I misconstrued your earlier remarks on cluster size vs number of clusters. As t -> 1 you will get fewer and fewer canopies as you have observed. It actually doesn't seem like the cosine distance measure is working very well for you.
>
>Have you mentioned the size of your dictionary earlier? Perhaps
increasing the number of stop words that are rejected will
decrease the vector size and make clustering work better. This
seems like the curse of dimensionality at work.
>
>On 5/31/12 11:18 AM, Pat Ferrel wrote:
>
>Oops, misspoke. 0 good, 1 bad for clustering at least
>>For similarity 1 good 0 bad.
>>
>>One is a similarity value and the other a distance measure.
>>
>>But the primary question is how to get better canopies. I would
expect that as the distance t gets small the number of canopies
gets large which is what I see in the data below. Jeff suggests
I try much smaller t to get less canopies and I will though I
don't understand the logic. The docs are not all that similar.
being from a general news crawl.
>>
>>When using the CosineDistanceMeasure in Canopy on a corpus of
150,000 docs I get:
>> t1 = t2 = 0.3 => 123094 canopies
>> t1 = t2 = 0.6 => 97035 canopies
>> t1 = t2 = 0.9 => 60160 canopies
>>
>>Obviously none of these values for t is very useful and it looks
like I need to make t even larger, which would seem to indicate
very loose/non-dense canopies, no? For very large ts are the
canopies useful?
>>
>>I'm trying both but the other odd thing is that it takes longer
to run canopy on this data than to run kmeans, a lot longer.
>>
>>On 5/31/12 12:44 AM, Sean Owen wrote:
>>
>>On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com> wrote:
>>>
>>>
>>>I see
>>>> double denominator = Math.sqrt(lengthSquaredp1) *
>>>>Math.sqrt(lengthSquaredp2);
>>>> // correct for floating-point rounding errors
>>>> if (denominator< dotProduct) {
>>>> denominator = dotProduct;
>>>> }
>>>> return 1.0 - dotProduct / denominator;
>>>>
>>>>So this is going to return 1 - cosine, right? So for
clustering the
>>>>distance 1 = very close, 0 = very far.
>>>>
>>>>
>>>>
When two vectors are close, the angle between them is small, so the cosine
>>>is large, near 1. 0 = close, 1 = far, as expected.
>>>
>>>
>>
>>
>
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Yeah, that's the conclusion I was coming to but thought I'd ask the
experts. My dictionary is petty big. the last time I looked it was over
100,000 terms even with n-grams, lucene stop words, no numbers, and
stemming. I've tried Tanimoto too with similar results.
Dimensional reduction seems like the next thing to try.
-Pat
Further data from 150,000 docs. Using Canopy clustering I get these values
t1 = t2 = 0.3 => 123094 canopies
t1 = t2 = 0.6 => 97035 canopies
t1 = t2 = 0.9 => 60160 canopies
t1 = t2 = 0.91 => 59491 canopies
t1 = t2 = 0.93 => 58526 canopies
t1 = t2 = 0.95 => 57854 canopies
t1 = t2 = 0.97 => 57244 canopies
t1 = t2 = 0.99 => 56241 canopies
On 5/31/12 2:31 PM, Jeff Eastman wrote:
> And I misconstrued your earlier remarks on cluster size vs number of
> clusters. As t -> 1 you will get fewer and fewer canopies as you have
> observed. It actually doesn't seem like the cosine distance measure is
> working very well for you.
>
> Have you mentioned the size of your dictionary earlier? Perhaps
> increasing the number of stop words that are rejected will decrease
> the vector size and make clustering work better. This seems like the
> curse of dimensionality at work.
>
> On 5/31/12 11:18 AM, Pat Ferrel wrote:
>> Oops, misspoke. 0 good, 1 bad for clustering at least
>> For similarity 1 good 0 bad.
>>
>> One is a similarity value and the other a distance measure.
>>
>> But the primary question is how to get better canopies. I would
>> expect that as the distance t gets small the number of canopies gets
>> large which is what I see in the data below. Jeff suggests I try much
>> smaller t to get less canopies and I will though I don't understand
>> the logic. The docs are not all that similar. being from a general
>> news crawl.
>>
>> When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
>> docs I get:
>> t1 = t2 = 0.3 => 123094 canopies
>> t1 = t2 = 0.6 => 97035 canopies
>> t1 = t2 = 0.9 => 60160 canopies
>>
>> Obviously none of these values for t is very useful and it looks like
>> I need to make t even larger, which would seem to indicate very
>> loose/non-dense canopies, no? For very large ts are the canopies useful?
>>
>> I'm trying both but the other odd thing is that it takes longer to
>> run canopy on this data than to run kmeans, a lot longer.
>>
>> On 5/31/12 12:44 AM, Sean Owen wrote:
>>> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com>
>>> wrote:
>>>
>>>> I see
>>>> double denominator = Math.sqrt(lengthSquaredp1) *
>>>> Math.sqrt(lengthSquaredp2);
>>>> // correct for floating-point rounding errors
>>>> if (denominator< dotProduct) {
>>>> denominator = dotProduct;
>>>> }
>>>> return 1.0 - dotProduct / denominator;
>>>>
>>>> So this is going to return 1 - cosine, right? So for clustering the
>>>> distance 1 = very close, 0 = very far.
>>>>
>>>>
>>> When two vectors are close, the angle between them is small, so the
>>> cosine
>>> is large, near 1. 0 = close, 1 = far, as expected.
>>>
>>
>>
>
Re: Clustering a large crawl
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
And I misconstrued your earlier remarks on cluster size vs number of
clusters. As t -> 1 you will get fewer and fewer canopies as you have
observed. It actually doesn't seem like the cosine distance measure is
working very well for you.
Have you mentioned the size of your dictionary earlier? Perhaps
increasing the number of stop words that are rejected will decrease the
vector size and make clustering work better. This seems like the curse
of dimensionality at work.
On 5/31/12 11:18 AM, Pat Ferrel wrote:
> Oops, misspoke. 0 good, 1 bad for clustering at least
> For similarity 1 good 0 bad.
>
> One is a similarity value and the other a distance measure.
>
> But the primary question is how to get better canopies. I would expect
> that as the distance t gets small the number of canopies gets large
> which is what I see in the data below. Jeff suggests I try much
> smaller t to get less canopies and I will though I don't understand
> the logic. The docs are not all that similar. being from a general
> news crawl.
>
> When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
> docs I get:
> t1 = t2 = 0.3 => 123094 canopies
> t1 = t2 = 0.6 => 97035 canopies
> t1 = t2 = 0.9 => 60160 canopies
>
> Obviously none of these values for t is very useful and it looks like
> I need to make t even larger, which would seem to indicate very
> loose/non-dense canopies, no? For very large ts are the canopies useful?
>
> I'm trying both but the other odd thing is that it takes longer to run
> canopy on this data than to run kmeans, a lot longer.
>
> On 5/31/12 12:44 AM, Sean Owen wrote:
>> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com>
>> wrote:
>>
>>> I see
>>> double denominator = Math.sqrt(lengthSquaredp1) *
>>> Math.sqrt(lengthSquaredp2);
>>> // correct for floating-point rounding errors
>>> if (denominator< dotProduct) {
>>> denominator = dotProduct;
>>> }
>>> return 1.0 - dotProduct / denominator;
>>>
>>> So this is going to return 1 - cosine, right? So for clustering the
>>> distance 1 = very close, 0 = very far.
>>>
>>>
>> When two vectors are close, the angle between them is small, so the
>> cosine
>> is large, near 1. 0 = close, 1 = far, as expected.
>>
>
>
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@farfetchers.com>.
Oops, misspoke. 0 good, 1 bad for clustering at least
For similarity 1 good 0 bad.
One is a similarity value and the other a distance measure.
But the primary question is how to get better canopies. I would expect
that as the distance t gets small the number of canopies gets large
which is what I see in the data below. Jeff suggests I try much smaller
t to get less canopies and I will though I don't understand the logic.
The docs are not all that similar. being from a general news crawl.
When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
docs I get:
t1 = t2 = 0.3 => 123094 canopies
t1 = t2 = 0.6 => 97035 canopies
t1 = t2 = 0.9 => 60160 canopies
Obviously none of these values for t is very useful and it looks like I
need to make t even larger, which would seem to indicate very
loose/non-dense canopies, no? For very large ts are the canopies useful?
I'm trying both but the other odd thing is that it takes longer to run
canopy on this data than to run kmeans, a lot longer.
On 5/31/12 12:44 AM, Sean Owen wrote:
> On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel<pa...@occamsmachete.com> wrote:
>
>> I see
>> double denominator = Math.sqrt(lengthSquaredp1) *
>> Math.sqrt(lengthSquaredp2);
>> // correct for floating-point rounding errors
>> if (denominator< dotProduct) {
>> denominator = dotProduct;
>> }
>> return 1.0 - dotProduct / denominator;
>>
>> So this is going to return 1 - cosine, right? So for clustering the
>> distance 1 = very close, 0 = very far.
>>
>>
> When two vectors are close, the angle between them is small, so the cosine
> is large, near 1. 0 = close, 1 = far, as expected.
>
Re: Clustering a large crawl
Posted by Sean Owen <sr...@gmail.com>.
On Thu, May 31, 2012 at 12:36 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> I see
> double denominator = Math.sqrt(lengthSquaredp1) *
> Math.sqrt(lengthSquaredp2);
> // correct for floating-point rounding errors
> if (denominator < dotProduct) {
> denominator = dotProduct;
> }
> return 1.0 - dotProduct / denominator;
>
> So this is going to return 1 - cosine, right? So for clustering the
> distance 1 = very close, 0 = very far.
>
>
When two vectors are close, the angle between them is small, so the cosine
is large, near 1. 0 = close, 1 = far, as expected.
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
I see
double denominator = Math.sqrt(lengthSquaredp1) *
Math.sqrt(lengthSquaredp2);
// correct for floating-point rounding errors
if (denominator < dotProduct) {
denominator = dotProduct;
}
return 1.0 - dotProduct / denominator;
So this is going to return 1 - cosine, right? So for clustering the
distance 1 = very close, 0 = very far.
When using the CosineDistanceMeasure in Canopy on a corpus of 150,000
docs I get:
t1 = t2 = 0.3 => 123094 canopies
t1 = t2 = 0.6 => 97035 canopies
t1 = t2 = 0.9 => 60160 canopies
The number of canopies seems to go down as t goes up so I assumed t was
actually a cosine in this case. I'd expect to get 150,000 canopies with
smaller t values. I have double checked and I did indeed use
org.apache.mahout.common.distance.CosineDistanceMeasure.
On 5/30/12 1:26 PM, Jeff Eastman wrote:
> The CosineDistanceMeasure returns 1 - dotProduct / denominator so it
> is returning the value you note. If the documents are very similar,
> then their distance will be small and t=0.1 could be too large to
> distinguish anything but the gross differences between the documents
> in the corpus. I'd try dropping the t-value until I get at least
> 50-100 clusters but I have no idea how small that might be.
>
>
> On 5/30/12 4:11 PM, Robert Stewart wrote:
>> That is a good point. t1/t2 are distance measures but cosine is a
>> similarity measure, so you need to think of it as 1-cosine.
>>
>>
>>
>> On May 30, 2012, at 4:03 PM, Jeff Eastman wrote:
>>
>>> Have you tried much smaller values for t1=t2? Recall that the
>>> t-values specify the distance within which a new point is assigned
>>> to an existing canopy. In the limit as t -> 0, you should get n
>>> clusters, where n is the number of documents in your corpus.
>>>
>>> On 5/30/12 1:23 PM, Pat Ferrel wrote:
>>>> I have about 150,000 docs on which I ran canopy with values for t1
>>>> = t2 from 0.1 to 0.95 using the Cosine distance measure. I got
>>>> results that range from 1.5 docs per cluster to 3. In other words
>>>> canopy produced a very large number of centroids, which does not
>>>> seem to represent the data very well. Trying random values for k
>>>> seems to produce better results but still spotty and hard to judge.
>>>> I am at the point of giving up on canopy and so wrote a utility to
>>>> simply iterate k over some values and run the evaluators each time,
>>>> but there are currently some problems with CDbw (Inter-Cluster
>>>> Density is always 0.0 for instance).
>>>>
>>>> This seems like such a fundamental problem that others must have
>>>> found a way to get better results. Any suggestions?
>>>>
>>>>
>>
>>
>
Re: RowSimilarityJob
Posted by Suneel Marthi <su...@yahoo.com>.
To answer ur question Pat, for SIMILARITY_COSINE the value returned = cosine.
________________________________
From: Suneel Marthi <su...@yahoo.com>
To: "user@mahout.apache.org" <us...@mahout.apache.org>
Sent: Wednesday, May 30, 2012 11:22 PM
Subject: Re: RowSimilarityJob
Pat,
Here is an example from the output of the rowsimilarity job for a corpus I am working with (using Cosine Similarity).
Key: 25: Value: {27433:0.9999999999999994}
What this means is that Document# 26 is similar to Document# 27433by a factor of 0.999.
Since Distance = (1 - Similarity), this means that the distance between documents 25 and 27433 above is 0 (= 1 - 0.999), or in other words they are very similar.
Hope that clarifies.
Suneel
________________________________
From: Pat Ferrel <pa...@occamsmachete.com>
To: user@mahout.apache.org
Sent: Wednesday, May 30, 2012 10:22 PM
Subject: RowSimilarityJob
What is the value created to describe simlarity by RowSimilarityJob? The paper which describes how the algorithm is implemented doesn't describe the various similarity values returned by mahout. It seems to focus on cooccurrences.
For SIMILARITY_COSINE is the value = cosine or 1 - cosine?
Is the value calculated after cooccurrences determines similar docs independently?
The code is very difficult to read so a little help would be appreciated.
Re: RowSimilarityJob
Posted by Suneel Marthi <su...@yahoo.com>.
Pat,
Here is an example from the output of the rowsimilarity job for a corpus I am working with (using Cosine Similarity).
Key: 25: Value: {27433:0.9999999999999994}
What this means is that Document# 26 is similar to Document# 27433by a factor of 0.999.
Since Distance = (1 - Similarity), this means that the distance between documents 25 and 27433 above is 0 (= 1 - 0.999), or in other words they are very similar.
Hope that clarifies.
Suneel
________________________________
From: Pat Ferrel <pa...@occamsmachete.com>
To: user@mahout.apache.org
Sent: Wednesday, May 30, 2012 10:22 PM
Subject: RowSimilarityJob
What is the value created to describe simlarity by RowSimilarityJob? The paper which describes how the algorithm is implemented doesn't describe the various similarity values returned by mahout. It seems to focus on cooccurrences.
For SIMILARITY_COSINE is the value = cosine or 1 - cosine?
Is the value calculated after cooccurrences determines similar docs independently?
The code is very difficult to read so a little help would be appreciated.
RowSimilarityJob
Posted by Pat Ferrel <pa...@occamsmachete.com>.
What is the value created to describe simlarity by RowSimilarityJob? The
paper which describes how the algorithm is implemented doesn't describe
the various similarity values returned by mahout. It seems to focus on
cooccurrences.
For SIMILARITY_COSINE is the value = cosine or 1 - cosine?
Is the value calculated after cooccurrences determines similar docs
independently?
The code is very difficult to read so a little help would be appreciated.
Re: Clustering a large crawl
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
The CosineDistanceMeasure returns 1 - dotProduct / denominator so it is
returning the value you note. If the documents are very similar, then
their distance will be small and t=0.1 could be too large to distinguish
anything but the gross differences between the documents in the corpus.
I'd try dropping the t-value until I get at least 50-100 clusters but I
have no idea how small that might be.
On 5/30/12 4:11 PM, Robert Stewart wrote:
> That is a good point. t1/t2 are distance measures but cosine is a similarity measure, so you need to think of it as 1-cosine.
>
>
>
> On May 30, 2012, at 4:03 PM, Jeff Eastman wrote:
>
>> Have you tried much smaller values for t1=t2? Recall that the t-values specify the distance within which a new point is assigned to an existing canopy. In the limit as t -> 0, you should get n clusters, where n is the number of documents in your corpus.
>>
>> On 5/30/12 1:23 PM, Pat Ferrel wrote:
>>> I have about 150,000 docs on which I ran canopy with values for t1 = t2 from 0.1 to 0.95 using the Cosine distance measure. I got results that range from 1.5 docs per cluster to 3. In other words canopy produced a very large number of centroids, which does not seem to represent the data very well. Trying random values for k seems to produce better results but still spotty and hard to judge. I am at the point of giving up on canopy and so wrote a utility to simply iterate k over some values and run the evaluators each time, but there are currently some problems with CDbw (Inter-Cluster Density is always 0.0 for instance).
>>>
>>> This seems like such a fundamental problem that others must have found a way to get better results. Any suggestions?
>>>
>>>
>
>
Re: Clustering a large crawl
Posted by Robert Stewart <bs...@gmail.com>.
That is a good point. t1/t2 are distance measures but cosine is a similarity measure, so you need to think of it as 1-cosine.
On May 30, 2012, at 4:03 PM, Jeff Eastman wrote:
> Have you tried much smaller values for t1=t2? Recall that the t-values specify the distance within which a new point is assigned to an existing canopy. In the limit as t -> 0, you should get n clusters, where n is the number of documents in your corpus.
>
> On 5/30/12 1:23 PM, Pat Ferrel wrote:
>> I have about 150,000 docs on which I ran canopy with values for t1 = t2 from 0.1 to 0.95 using the Cosine distance measure. I got results that range from 1.5 docs per cluster to 3. In other words canopy produced a very large number of centroids, which does not seem to represent the data very well. Trying random values for k seems to produce better results but still spotty and hard to judge. I am at the point of giving up on canopy and so wrote a utility to simply iterate k over some values and run the evaluators each time, but there are currently some problems with CDbw (Inter-Cluster Density is always 0.0 for instance).
>>
>> This seems like such a fundamental problem that others must have found a way to get better results. Any suggestions?
>>
>>
>
Re: Clustering a large crawl
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Have you tried much smaller values for t1=t2? Recall that the t-values
specify the distance within which a new point is assigned to an existing
canopy. In the limit as t -> 0, you should get n clusters, where n is
the number of documents in your corpus.
On 5/30/12 1:23 PM, Pat Ferrel wrote:
> I have about 150,000 docs on which I ran canopy with values for t1 =
> t2 from 0.1 to 0.95 using the Cosine distance measure. I got results
> that range from 1.5 docs per cluster to 3. In other words canopy
> produced a very large number of centroids, which does not seem to
> represent the data very well. Trying random values for k seems to
> produce better results but still spotty and hard to judge. I am at the
> point of giving up on canopy and so wrote a utility to simply iterate
> k over some values and run the evaluators each time, but there are
> currently some problems with CDbw (Inter-Cluster Density is always 0.0
> for instance).
>
> This seems like such a fundamental problem that others must have found
> a way to get better results. Any suggestions?
>
>
Re: Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
Using:
* A random crawl starting from mostly news sites.
* TFIDF
* a custom lucene analyzer with stemming, stop words, lowercasing,
number removal
* tried bi-grams but didn't like the results, tried large mll of 2000
but still got too many meaningless ones, will revisit but not all
that important I think.
* L2 normalization
I know it has clusters because when I use a random k value I get a lot
of good clusters and quite a few bad ones.
On 5/30/12 10:39 AM, Robert Stewart wrote:
> What type of documents (news, web, pdf, etc.)? How are your vectors constructed? Are you using TF*IDF on just uni-terms or n-grams, etc? Do you stop or stem the content? How do you know the data contains the type of clusters you expect mahout to find?
>
> Some suggestions:
> Make sure you remove stop words and use stemming on remaining terms
> Try using bi-grams instead of single terms
> Try building clusters from document headlines only (in the case of news articles)
>
>
>
>
>
> On May 30, 2012, at 1:23 PM, Pat Ferrel wrote:
>
>> I have about 150,000 docs on which I ran canopy with values for t1 = t2 from 0.1 to 0.95 using the Cosine distance measure. I got results that range from 1.5 docs per cluster to 3. In other words canopy produced a very large number of centroids, which does not seem to represent the data very well. Trying random values for k seems to produce better results but still spotty and hard to judge. I am at the point of giving up on canopy and so wrote a utility to simply iterate k over some values and run the evaluators each time, but there are currently some problems with CDbw (Inter-Cluster Density is always 0.0 for instance).
>>
>> This seems like such a fundamental problem that others must have found a way to get better results. Any suggestions?
>
>
Re: Clustering a large crawl
Posted by Robert Stewart <bs...@gmail.com>.
What type of documents (news, web, pdf, etc.)? How are your vectors constructed? Are you using TF*IDF on just uni-terms or n-grams, etc? Do you stop or stem the content? How do you know the data contains the type of clusters you expect mahout to find?
Some suggestions:
Make sure you remove stop words and use stemming on remaining terms
Try using bi-grams instead of single terms
Try building clusters from document headlines only (in the case of news articles)
On May 30, 2012, at 1:23 PM, Pat Ferrel wrote:
> I have about 150,000 docs on which I ran canopy with values for t1 = t2 from 0.1 to 0.95 using the Cosine distance measure. I got results that range from 1.5 docs per cluster to 3. In other words canopy produced a very large number of centroids, which does not seem to represent the data very well. Trying random values for k seems to produce better results but still spotty and hard to judge. I am at the point of giving up on canopy and so wrote a utility to simply iterate k over some values and run the evaluators each time, but there are currently some problems with CDbw (Inter-Cluster Density is always 0.0 for instance).
>
> This seems like such a fundamental problem that others must have found a way to get better results. Any suggestions?
Clustering a large crawl
Posted by Pat Ferrel <pa...@occamsmachete.com>.
I have about 150,000 docs on which I ran canopy with values for t1 = t2
from 0.1 to 0.95 using the Cosine distance measure. I got results that
range from 1.5 docs per cluster to 3. In other words canopy produced a
very large number of centroids, which does not seem to represent the
data very well. Trying random values for k seems to produce better
results but still spotty and hard to judge. I am at the point of giving
up on canopy and so wrote a utility to simply iterate k over some values
and run the evaluators each time, but there are currently some problems
with CDbw (Inter-Cluster Density is always 0.0 for instance).
This seems like such a fundamental problem that others must have found a
way to get better results. Any suggestions?
Re: Continue K-Means on pseudo-distributed mode after a server failure
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Why not run in sequential mode (-xm sequential) and save all the Hadoop
overhead?
On 5/24/12 8:29 AM, Michael Kazekin wrote:
> Hi!
>
> I have run a K-Means on a single node (pseudo-distributed mode) and
> experienced a server failure.
>
> Is there any way to continue clustering with existing M/R jobs?
>
> Cheers,
> Michael.
>
>