You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Owen Densmore <ow...@backspaces.net> on 2005/02/23 16:31:49 UTC
Term Weights and Clustering
I'm building a TDM (Term Document Matrix) from my lucene index. As
part of this, it would be useful to have the document term weights (the
TF*IDF-weight) if they are already available. Naturally I can compute
them, but I suspect they are lurking behind an API I've not discovered
yet. Is there an API for getting them?
I'm doing this as a first step in discovering a good set of clustering
labels. My data collection is 1200 research papers, all of which have
good meta data: titles, authors, abstracts, keyphrases and so on.
One source for how to do this is the thesis of Stanislaw Osinski and
others like it:
http://www.dcs.shef.ac.uk/teaching/eproj/msc2004/abs/m3so.htm
And the Carrot2 project which uses similar techniques.
http://www.cs.put.poznan.pl/dweiss/carrot/
My problem is simple: I need a fairly clear discussion on exactly how
to generate the labels, and to assign documents to them. The thesis is
quite good, but I'm not sure I can reduce it to practice in the 2-3
days I have to evaluate it! Lucene has made the TDM easy to calculate,
but I basically don't know what to do next!
Can anyone comment on whether or not this will work, and if so, suggest
a quick way to get a demo on the air? For example, I don't seem to be
able to ask Carrot2 to do a Google "site" search. If I could, I could
simply aim Carrot2 at my collection with a very general search and see
what clusters it discovers. This may be a gross misuse of Carrot2's
clustering anyway, so could easily be a blind alley.
Or is there a different stunt with Lucene that might work? For
example, use Lucene to cluster the docs using a batch search where the
queries are Library of Congress descriptions! Batch searching is
*really fast* in Lucene -- I've been able to search the data collection
against each distinct keyphrase in seconds!
Owen
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Term Weights and Clustering
Posted by David Spencer <da...@tropo.com>.
I'm a little confused on exactly, exactly what you want but if your goal
is to cluster your papers w/ carrot2 then I found these links helpful:
http://www.newsarch.com/archive/mailinglist/jakarta/lucene/user/msg03928.html
http://www.cs.put.poznan.pl/dweiss/tmp/carrot2-lucene.zip
Only caveat is I found that carrot2 tends to not scale beyond 200 or so
docs, though this probably depends on length of docs & the # of
different tokens.
I was able to use the above to integ w/ a lucene search results page in
just an hour or so.
Owen Densmore wrote:
> I'm building a TDM (Term Document Matrix) from my lucene index. As part
> of this, it would be useful to have the document term weights (the
> TF*IDF-weight) if they are already available. Naturally I can compute
> them, but I suspect they are lurking behind an API I've not discovered
> yet. Is there an API for getting them?
>
> I'm doing this as a first step in discovering a good set of clustering
> labels. My data collection is 1200 research papers, all of which have
> good meta data: titles, authors, abstracts, keyphrases and so on.
>
> One source for how to do this is the thesis of Stanislaw Osinski and
> others like it:
> http://www.dcs.shef.ac.uk/teaching/eproj/msc2004/abs/m3so.htm
> And the Carrot2 project which uses similar techniques.
> http://www.cs.put.poznan.pl/dweiss/carrot/
>
> My problem is simple: I need a fairly clear discussion on exactly how to
> generate the labels, and to assign documents to them. The thesis is
> quite good, but I'm not sure I can reduce it to practice in the 2-3 days
> I have to evaluate it! Lucene has made the TDM easy to calculate, but I
> basically don't know what to do next!
>
> Can anyone comment on whether or not this will work, and if so, suggest
> a quick way to get a demo on the air? For example, I don't seem to be
> able to ask Carrot2 to do a Google "site" search. If I could, I could
> simply aim Carrot2 at my collection with a very general search and see
> what clusters it discovers. This may be a gross misuse of Carrot2's
> clustering anyway, so could easily be a blind alley.
>
> Or is there a different stunt with Lucene that might work? For example,
> use Lucene to cluster the docs using a batch search where the queries
> are Library of Congress descriptions! Batch searching is *really fast*
> in Lucene -- I've been able to search the data collection against each
> distinct keyphrase in seconds!
>
> Owen
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Term Weights and Clustering
Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
Hi Owen,
I'm from the Carrot2 project, so I feel called to the blackboard:
> One source for how to do this is the thesis of Stanislaw Osinski and
> others like it:
> http://www.dcs.shef.ac.uk/teaching/eproj/msc2004/abs/m3so.htm
> And the Carrot2 project which uses similar techniques.
> http://www.cs.put.poznan.pl/dweiss/carrot/
Staszek Osinski is the author of Lingo, the best clustering algorithm
available in Carrot2 -- we still work together in that project... In
other words, Carrot2 doesn't use 'similar' techniques. It uses _the_
techniques described in the above thesis (and other various papars, see
my Web page).
> My problem is simple: I need a fairly clear discussion on exactly how to
> generate the labels, and to assign documents to them. The thesis is
> quite good, but I'm not sure I can reduce it to practice in the 2-3 days
> I have to evaluate it! Lucene has made the TDM easy to calculate, but I
> basically don't know what to do next!
You can use Carrot2 directly for that. There are a few options. One
thing is you can directly feed your input collection to the clustering
component (it will take a while, but should work) -- you need to write a
custom input component, but it is a very simple thing to do and I'm sure
if you write to Carrot2 mailing list there will be somebody willing to
help (like myself or Staszek ;).
Another option is: use Lucene to index your documents. Set up Carrot2 to
use Lucene (described somewhere on this list, see David Spencer's message).
> a quick way to get a demo on the air? For example, I don't seem to be
> able to ask Carrot2 to do a Google "site" search.
Yep, there is a problem with it. Post a bug report to carrot2 bugzilla,
please. I'll investigate it when I have time.
> simply aim Carrot2 at my collection with a very general search and see
> what clusters it discovers. This may be a gross misuse of Carrot2's
> clustering anyway, so could easily be a blind alley.
It kind of is because carrot2 clustering components work primarily with
_short_, scarce information sources, such as snippets. We don't intend
to work on large, raw documents collections... Having said that, a 1200
documents isn't that much and you should be able to get your clusters.
D.
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org