Re: Document Clustering

[Owen Densmore <ow...@backspaces.net>]

I would like to be able to analyze my document collection (~1200 
documents) and discover good "buckets" of categories for them.  I'm 
pretty sure this is termed Document Clustering .. finding the emergent 
clumps the documents fall naturally into judging from their term 
vectors.

Looking at the discussion that flared roughly a year ago (last message 
2003-11-12) with the subject Document Clustering, it seems Lucene 
should be able to help with this.  Has anyone had success with this 
recently?

Last year it was suggested Carrot2 could help, and it would even 
produce good labels for the clusters.  Has this proven to be true?  Our 
goal is to use clustering to build a nifty graphic interface, probably 
using Flash.

Thanks for any pointers.

Owen


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Document Clustering

[David Spencer <da...@tropo.com>]

Owen Densmore wrote:

> I would like to be able to analyze my document collection (~1200 
> documents) and discover good "buckets" of categories for them.  I'm 
> pretty sure this is termed Document Clustering .. finding the emergent 
> clumps the documents fall naturally into judging from their term vectors.
> 
> Looking at the discussion that flared roughly a year ago (last message 
> 2003-11-12) with the subject Document Clustering, it seems Lucene should 
> be able to help with this.  Has anyone had success with this recently?
> 
> Last year it was suggested Carrot2 could help, and it would even produce 
> good labels for the clusters.  Has this proven to be true?  Our goal is 
> to use clustering to build a nifty graphic interface, probably using Flash.

Carrot2 seems to work nicely.
Demo here...

Search for something like "artificial intelligence" in my Wikipedia 
Search engine:

http://www.searchmorph.com/kat/wikipedia.jsp?s=artificial+intelligence

The click on "see clustered results.." link to go here:

http://www.searchmorph.com/kat/wikipedia-cluster.jsp?s=artificial%20intelligence

And voilla, what seems like decent clusters.

I'm not sure what the complexity of the algorithm is, but for me ~100 
docs works ok, maybe 200, but beyond 200 you need lots more CPU and RAM.

I suggest: try it w/ ~100 docs, and if you like what you see, keep 
increasing the # of docs you give it. You might have to wait a while w/ 
all 1,200 docs...

- Dave






> 
> Thanks for any pointers.
> 
> Owen
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Document Clustering

[Dawid Weiss <da...@cs.put.poznan.pl>]

Hi Owen,

> Last year it was suggested Carrot2 could help, and it would even produce 
> good labels for the clusters.  Has this proven to be true?  

Yes, Carrot2 should help you with this. The labels it creates highly 
depend on the quality of the input snippets, but the so-called KWIK 
snippets (keyword in context) should suffice (see David Spencer's 
example with Wikipedia).

There is one thing, though: what is employed in Carrot2 is an on-line 
unsupervised clusterer that is designed to work with small number of 
documents and incomplete descriptions (snippets versus full text 
documents). It will _not_ work for large document collections (thousands 
of documents) simply because it was not designed to do that. I guess
you could try with up to 500 snippets -- beyond that, you'll be waiting 
for the result forever.

There is a great number of algorithms that can cluster large document 
collections -- see proceedings from information retrieval conferences 
for example.

As for David's hints:

 > I'm not sure what the complexity of the algorithm is, but for me ~100 
 > docs works ok, maybe 200, but beyond 200 you need lots more CPU and RAM.

Yes, 100 to 200 snippets is optimal with the open source clustering 
algorithm. We have a refactored and optimized version of the Lingo 
clusterer that is commercial (it also provides hierarchical clustering 
capability as an add-on to the open source component). But even the 
commercial version will only cluster up to 500 -- 1000 snippets. As I 
said, it was not our goal to cluster document collections, rather to 
retrieve useful information from preprocessed snippets.

Dawid


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org