You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Grant Ingersoll <gs...@apache.org> on 2009/06/17 13:16:25 UTC
[Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse vectors
Shashikant asked this over on mahout-dev, but I thought I would move
it to user so that others can benefit from the discussion.
On Jun 17, 2009, at 1:12 AM, Shashikant Kore (JIRA) wrote:
>
> Shashikant Kore commented on MAHOUT-121:
> ----------------------------------------
>
>
> [OT] Also, was wondering how you came up with the values of t1 and
> t2 as 1.3 & 1.0. This is voodoo for me. For the dataset I am
> working with has a window of 0.05 in which the result changes from 0
> canopies to 3,000 canopies.
I just picked some numbers based on what you did! It is voodoo to me
too. I have not done much clustering, so I'm learning a lot here. As
for MAHOUT-121, I just wanted something to run.
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse vectors
Posted by Daniel Kulp <dk...@apache.org>.
It depends on the purpose of the confluence space. If the space is used at
all for "official" project related material, an ICLA must be on file.
"official" project related material includes (but is not limitted to):
1) The project web site
2) Anything going into a "release"
3) Any "official" documentation
Documentation, on a wiki or in SVN, is a project owned resource and MUST have
the same legal protection as the source code. Thus, the ICLA must be on file
for it.
If the space is just for discussion things and similar, then no, an ICLA would
not be required.
Dan
On Wed June 17 2009 10:38:41 am Benson Margulies wrote:
> Grant,
>
> That's not the policy as we've had it explained to us at CXF. If it's
> content on www.apache.org, any contributor has to have a CLA.
>
> Dan, do I have this right? Is there some difference between Confluence
> and cwiki?
>
> --benson
>
> On Wed, Jun 17, 2009 at 9:35 AM, Grant Ingersoll<gs...@apache.org> wrote:
> > On Jun 17, 2009, at 9:29 AM, Benson Margulies wrote:
> >> At CXF, permission to modify the confluence wiki is only granted to
> >> people with a CLA on file? Obviously, I have one, but do you need to
> >> grant me karma
> >> here before I can edit?
> >
> > We aren't distributing the wiki as a part of the release, so anyone with
> > an account should be able to modify it.
--
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog
Re: Confluence as a Wiki was Re: [Canopy] Picking t1 and t2 was Re:
[jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse
vectors
Posted by Benson Margulies <bi...@gmail.com>.
As per Dan's email, all is well. Sorry for the faux alarm.
On Wed, Jun 17, 2009 at 10:49 AM, Grant Ingersoll <gs...@apache.org>wrote:
> I'm basing my understanding on reading http://cwiki.apache.org/CWIKI/. We
> are not using the Wiki as our main site nor are we packaging it with our
> releases and it is clearly labeled as a Wiki. It is effectively the same as
> any other Moin Moin Wiki in Apache land, AIUI.
>
> On Jun 17, 2009, at 10:38 AM, Benson Margulies wrote:
>
> Grant,
>>
>> That's not the policy as we've had it explained to us at CXF. If it's
>> content on www.apache.org, any contributor has to have a CLA.
>>
>> Dan, do I have this right? Is there some difference between Confluence
>> and cwiki?
>>
>> --benson
>>
>>
>> On Wed, Jun 17, 2009 at 9:35 AM, Grant Ingersoll<gs...@apache.org>
>> wrote:
>>
>>>
>>> On Jun 17, 2009, at 9:29 AM, Benson Margulies wrote:
>>>
>>> At CXF, permission to modify the confluence wiki is only granted to
>>>> people
>>>> with a CLA on file? Obviously, I have one, but do you need to grant me
>>>> karma
>>>> here before I can edit?
>>>>
>>>
>>> We aren't distributing the wiki as a part of the release, so anyone with
>>> an
>>> account should be able to modify it.
>>>
>>>
>>>
>
>
Confluence as a Wiki was Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse vectors
Posted by Grant Ingersoll <gs...@apache.org>.
I'm basing my understanding on reading http://cwiki.apache.org/
CWIKI/. We are not using the Wiki as our main site nor are we
packaging it with our releases and it is clearly labeled as a Wiki.
It is effectively the same as any other Moin Moin Wiki in Apache land,
AIUI.
On Jun 17, 2009, at 10:38 AM, Benson Margulies wrote:
> Grant,
>
> That's not the policy as we've had it explained to us at CXF. If it's
> content on www.apache.org, any contributor has to have a CLA.
>
> Dan, do I have this right? Is there some difference between Confluence
> and cwiki?
>
> --benson
>
>
> On Wed, Jun 17, 2009 at 9:35 AM, Grant
> Ingersoll<gs...@apache.org> wrote:
>>
>> On Jun 17, 2009, at 9:29 AM, Benson Margulies wrote:
>>
>>> At CXF, permission to modify the confluence wiki is only granted
>>> to people
>>> with a CLA on file? Obviously, I have one, but do you need to
>>> grant me
>>> karma
>>> here before I can edit?
>>
>> We aren't distributing the wiki as a part of the release, so anyone
>> with an
>> account should be able to modify it.
>>
>>
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Benson Margulies <bi...@gmail.com>.
Grant,
That's not the policy as we've had it explained to us at CXF. If it's
content on www.apache.org, any contributor has to have a CLA.
Dan, do I have this right? Is there some difference between Confluence
and cwiki?
--benson
On Wed, Jun 17, 2009 at 9:35 AM, Grant Ingersoll<gs...@apache.org> wrote:
>
> On Jun 17, 2009, at 9:29 AM, Benson Margulies wrote:
>
>> At CXF, permission to modify the confluence wiki is only granted to people
>> with a CLA on file? Obviously, I have one, but do you need to grant me
>> karma
>> here before I can edit?
>
> We aren't distributing the wiki as a part of the release, so anyone with an
> account should be able to modify it.
>
>
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse vectors
Posted by Grant Ingersoll <gs...@apache.org>.
On Jun 17, 2009, at 9:29 AM, Benson Margulies wrote:
> At CXF, permission to modify the confluence wiki is only granted to
> people
> with a CLA on file? Obviously, I have one, but do you need to grant
> me karma
> here before I can edit?
We aren't distributing the wiki as a part of the release, so anyone
with an account should be able to modify it.
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Benson Margulies <bi...@gmail.com>.
At CXF, permission to modify the confluence wiki is only granted to people
with a CLA on file? Obviously, I have one, but do you need to grant me karma
here before I can edit?
On Wed, Jun 17, 2009 at 9:22 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Jun 17, 2009, at 9:05 AM, Benson Margulies wrote:
>
> All I know is what I learned from reading the paper. However, I continue
>> to
>> think, from reading the paper, that you may be trying to make Canopy do
>> something it was not intended to do.
>>
>> As I read the paper, the idea here is to get a rough partitioning that is
>> used to optimize various downstream algorithms, not to tune for a precise
>> partitioning. The number of canopies doesn't need, as I read it, to be
>> particularly close to the number of eventual partitions to be useful.
>>
>> Thus the extended discussion of how to start up and run various other
>> algorithms, (e.g. k-means).
>>
>
> Makes sense.
>
>
>> Now, still, you need to get some useful number of partitions. The paper
>> has
>> a classic toss-off line, 'we used cross-validation,' without any details
>> about exactly what the authors did. Presumably, that means that the author
>> ran many possible values and hand-examined the results. The paper reports
>> no
>> general results about how sensitive the T values are to particular input
>> data sets. A pessimist would fear that, for any new input, you're going to
>> need to go through a lengthy process to find good values for T1 and T2.
>>
>> This leads me to wonder, ignorantly, why this project is so focused on
>> Canopy. The paper describes it as a tool for speeding up various other
>> things. Since you're hadooping all those other things, how much does it
>> help?
>>
>
> I don't think anyone is solely focused on it, but it is something that we
> have available in our arsenal of clustering tools, therefore it warrants
> documentation and understanding of when and how to use. Personally, it's
> just something I could easily run to work on MAHOUT-121.
>
> At any rate, this kind of write up is exactly the advice that we need to be
> able to give people. Care to add to
> http://cwiki.apache.org/confluence/display/MAHOUT/ClusteringYourData ?
>
>
>
>> Anyway, I expect that my ignorance is on comprehensive display here.
>>
>>
> Funny, I feel like my ignorance is the one on display, but that is
> something I got over a long time ago in open source. Which is why I just
> come out and ask the questions! One of my goals for Mahout is to make it a
> place where people can come and learn about Machine Learning and get
> practical advice and not be afraid to ask basic questions. Machine learning
> is so shrouded in mystery it almost seems like a Dark Art. I'm thankful
> every day on this project that smarter people than me show up and answer
> questions. So, please, keep 'em coming!
>
> -Grant
>
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse vectors
Posted by Grant Ingersoll <gs...@apache.org>.
On Jun 17, 2009, at 9:05 AM, Benson Margulies wrote:
> All I know is what I learned from reading the paper. However, I
> continue to
> think, from reading the paper, that you may be trying to make Canopy
> do
> something it was not intended to do.
>
> As I read the paper, the idea here is to get a rough partitioning
> that is
> used to optimize various downstream algorithms, not to tune for a
> precise
> partitioning. The number of canopies doesn't need, as I read it, to be
> particularly close to the number of eventual partitions to be useful.
>
> Thus the extended discussion of how to start up and run various other
> algorithms, (e.g. k-means).
Makes sense.
>
> Now, still, you need to get some useful number of partitions. The
> paper has
> a classic toss-off line, 'we used cross-validation,' without any
> details
> about exactly what the authors did. Presumably, that means that the
> author
> ran many possible values and hand-examined the results. The paper
> reports no
> general results about how sensitive the T values are to particular
> input
> data sets. A pessimist would fear that, for any new input, you're
> going to
> need to go through a lengthy process to find good values for T1 and
> T2.
>
> This leads me to wonder, ignorantly, why this project is so focused on
> Canopy. The paper describes it as a tool for speeding up various other
> things. Since you're hadooping all those other things, how much does
> it
> help?
I don't think anyone is solely focused on it, but it is something that
we have available in our arsenal of clustering tools, therefore it
warrants documentation and understanding of when and how to use.
Personally, it's just something I could easily run to work on
MAHOUT-121.
At any rate, this kind of write up is exactly the advice that we need
to be able to give people. Care to add to http://cwiki.apache.org/confluence/display/MAHOUT/ClusteringYourData
?
>
> Anyway, I expect that my ignorance is on comprehensive display here.
>
Funny, I feel like my ignorance is the one on display, but that is
something I got over a long time ago in open source. Which is why I
just come out and ask the questions! One of my goals for Mahout is to
make it a place where people can come and learn about Machine Learning
and get practical advice and not be afraid to ask basic questions.
Machine learning is so shrouded in mystery it almost seems like a Dark
Art. I'm thankful every day on this project that smarter people than
me show up and answer questions. So, please, keep 'em coming!
-Grant
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Shashikant Kore <sh...@gmail.com>.
Grant,
Well, it's the other way round. Bandhan.com is a personal project.
It's search engine for matrimony (dating) profile aggregator focused
on Indian market.
I am consulting to a company and the Mahout work is for the same. I am
not sure if I am at liberty to disclose the identity. Please accept my
apologies for that.
But I can say I am working with data which we all work with - PDF/Word
documents, emails, text files, etc. Just that it's at scale.
--shashi
On Wed, Jun 17, 2009 at 8:08 PM, Grant Ingersoll<gs...@apache.org> wrote:
>
> On Jun 17, 2009, at 10:34 AM, Shashikant Kore wrote:
>>
>> OK. Now, let's not focus on my ignorance. I have got my hands dirty
>> with Machine Learning, Mahout and Hadoop barely few days back.
>>
>> --shashi
>>
>> -- http://www.bandhan.com/
>
> Shashi, based on your link here, I'm curious if you could share what your
> use case is a bit more? Or is this just a personal thing? Feel free to say
> you can't, but I think it would be helpful to others.
>
--
http://www.bandhan.com/
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121) Speed up distance calculations for sparse vectors
Posted by Grant Ingersoll <gs...@apache.org>.
On Jun 17, 2009, at 10:34 AM, Shashikant Kore wrote:
> OK. Now, let's not focus on my ignorance. I have got my hands dirty
> with Machine Learning, Mahout and Hadoop barely few days back.
>
> --shashi
>
> --
> http://www.bandhan.com/
Shashi, based on your link here, I'm curious if you could share what
your use case is a bit more? Or is this just a personal thing? Feel
free to say you can't, but I think it would be helpful to others.
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Shashikant Kore <sh...@gmail.com>.
I have verifed the results only by "laugh-test" method. Many of the
clusters were excellent. There were some false-positives though, which
were farther from the cetroid. It might be because I used 4
iterations. Higher number of iterations probably will give better
results.
Right now, I don't have any visualization tools to make a confident
statement about quality of clusters. I will report back when I have
something concrete.
--shashi
On Thu, Jun 18, 2009 at 12:16 AM, Ted Dunning<te...@gmail.com> wrote:
> Shashi,
>
> What were the results for k-means?
>
> (I have zero experience with canopy, but have generally had mildly useful
> results using k-means clustering.
>
> On Wed, Jun 17, 2009 at 7:34 AM, Shashikant Kore <sh...@gmail.com>wrote:
>
>> I ran Canopy and then K-Means on 50k doc vectors
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Ted Dunning <te...@gmail.com>.
Shashi,
What were the results for k-means?
(I have zero experience with canopy, but have generally had mildly useful
results using k-means clustering.
On Wed, Jun 17, 2009 at 7:34 AM, Shashikant Kore <sh...@gmail.com>wrote:
> I ran Canopy and then K-Means on 50k doc vectors
>
--
Ted Dunning, CTO
DeepDyve
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Shashikant Kore <sh...@gmail.com>.
On Wed, Jun 17, 2009 at 6:35 PM, Benson Margulies<bi...@gmail.com> wrote:
>
> As I read the paper, the idea here is to get a rough partitioning that is
> used to optimize various downstream algorithms, not to tune for a precise
> partitioning. The number of canopies doesn't need, as I read it, to be
> particularly close to the number of eventual partitions to be useful.
>
> Thus the extended discussion of how to start up and run various other
> algorithms, (e.g. k-means).
>
That's right. But here is my experience. I ran Canopy and then
K-Means on 50k doc vectors. (That, by the way, is fraction of the
actual dataset.) I used the code in the patch of 121, which uses
primitives for Sparse Vectors.
After some experimentation, for the t2 value of 0.9, I got only 1
cluster. When I changed it to 0.85, it generated 3000+ clusters(or
canopies). With increasing number of canopies the code starts
crawling. And after some time, even 2G memory is not sufficient for
it.
Canopies is one of the simplest clustering algorithm and I had trouble
getting it work. May be it's my data set. I simply didn't had the
patience to find out all the values of t1 and t2 which are anyway
going to change when the input changes. So, for now, I have just put a
cap on the number of canopies generated. Not elegant, but results
don't seem bad at all.
OK. Now, let's not focus on my ignorance. I have got my hands dirty
with Machine Learning, Mahout and Hadoop barely few days back.
--shashi
--
http://www.bandhan.com/
Re: [Canopy] Picking t1 and t2 was Re: [jira] Commented: (MAHOUT-121)
Speed up distance calculations for sparse vectors
Posted by Benson Margulies <bi...@gmail.com>.
All I know is what I learned from reading the paper. However, I continue to
think, from reading the paper, that you may be trying to make Canopy do
something it was not intended to do.
As I read the paper, the idea here is to get a rough partitioning that is
used to optimize various downstream algorithms, not to tune for a precise
partitioning. The number of canopies doesn't need, as I read it, to be
particularly close to the number of eventual partitions to be useful.
Thus the extended discussion of how to start up and run various other
algorithms, (e.g. k-means).
Now, still, you need to get some useful number of partitions. The paper has
a classic toss-off line, 'we used cross-validation,' without any details
about exactly what the authors did. Presumably, that means that the author
ran many possible values and hand-examined the results. The paper reports no
general results about how sensitive the T values are to particular input
data sets. A pessimist would fear that, for any new input, you're going to
need to go through a lengthy process to find good values for T1 and T2.
This leads me to wonder, ignorantly, why this project is so focused on
Canopy. The paper describes it as a tool for speeding up various other
things. Since you're hadooping all those other things, how much does it
help?
Anyway, I expect that my ignorance is on comprehensive display here.
On Wed, Jun 17, 2009 at 7:16 AM, Grant Ingersoll <gs...@apache.org>wrote:
> Shashikant asked this over on mahout-dev, but I thought I would move it to
> user so that others can benefit from the discussion.
>
>
> On Jun 17, 2009, at 1:12 AM, Shashikant Kore (JIRA) wrote:
>
>>
>> Shashikant Kore commented on MAHOUT-121:
>> ----------------------------------------
>>
>>
>> [OT] Also, was wondering how you came up with the values of t1 and t2 as
>> 1.3 & 1.0. This is voodoo for me. For the dataset I am working with has a
>> window of 0.05 in which the result changes from 0 canopies to 3,000
>> canopies.
>>
>
> I just picked some numbers based on what you did! It is voodoo to me too.
> I have not done much clustering, so I'm learning a lot here. As for
> MAHOUT-121, I just wanted something to run.