You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nik Kolev <nk...@healthmarketscience.com> on 2010/08/15 23:01:31 UTC
cluster documents based on fields' values
Hi,
I am researching the possibility of using Lucene for discovering
clusters of documents and since I am new to Lucene I decided to
ask the community for advice before I poke the APIs and the internals.
Your input will be invaluable!
Here's the use case. Documents arrive from different feeds. Each feed
produces millions and millions of documents. Documents are structured
and share certain "interesting" fields. For the purpose of illustration
here's a trivial example. Let's say the documents on each feed represent
shoes. A shoe has an ID (uniquely identifying it within its feed) and a
SIZE [1]. I want to be able to ask Lucene (assuming the shoe documents
are indexed of course) for all of the clusters of shoes that sport the same
size. A cluster of shoes is just the IDs of the shoes that got grouped
together due to having the same value of the SIZE field.
I don't think that doing this brute force will perform. Here's what I
mean when I say brute force. I looked at IndexReader, and I saw that
I can get the distinct values for an indexed field. So assuming that
each feed will have its own index (I don't think I can get away with
a single index for all feeds [2]), I can get the union of all distinct
values for the interesting field across all indexes (one per each
document feed). Then for each distinct value I can do a MultiSearcher
search across these indexes getting the IDs of the documents .
My gut tells me that the brute force approach won't perform [3]. And
here's where you guys come in - is it possible to ask lucene to give
me the groups of records (across indixes) that share the value for a
given field? Is there something else in API that I can take advantage
of and get what I need faster? If I am to extend Lucene to allow for
this sort of thing, where should I start, what documents should I
read,...?
Thanks very much again for your advise,
-nik
[1] Shoe document layout:
SHOE
|--ID
|--SIZE
[2] The volume of documents I get from each feed is huge. Also, even
though a lot of fields are shared the format across feeds varies
and a feed may not have some of the "interesting" fields of
another feed.
[3] The dataset I am talking about is rich and the brute force approach
means that I need to be doing tens of millions of searches just to
group on one field. Also I most likely will blow my heap up if I try to
load all of the values in memory all at once.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: cluster documents based on fields' values
Posted by Stanislaw Osinski <st...@carrotsearch.com>.
>
> A colleague of mine also discovered solr's clustering component -
> http://wiki.apache.org/solr/ClusteringComponent. It's still labeled as
> experimental - does anybody have experience with it?
>
The clustering component is based on the Carrot2 project (
project.carrot2.org). Carrot2 has been around for many years and is a mature
piece of software. However, the scope of the clustering component is
currently limited to post-retrieval clustering, ie. clustering of search
results, not the whole index. If you're looking for large-scale clustering,
Mahout would be the way to go.
Cheers,
Stanislaw
RE: cluster documents based on fields' values
Posted by Nik Kolev <nk...@healthmarketscience.com>.
Thanks Grant. I'll take a look at Solr's faceting.
A colleague of mine also discovered solr's clustering component -
http://wiki.apache.org/solr/ClusteringComponent. It's still labeled as
experimental - does anybody have experience with it?
Another option (pointed out by your post:
http://www.lucidimagination.com/blog/2010/03/16/integrating-apache-mahou
t-with-apache-lucene-and-solr-part-i-of-3/) is mahout with its baked in
clustering algorithms. Has anybody gotten good mileage out of this
approach?
Thanks again,
-nik
-----Original Message-----
From: Grant Ingersoll [mailto:gsingers@apache.org]
Sent: Tuesday, August 17, 2010 7:57 AM
To: java-user@lucene.apache.org
Subject: Re: cluster documents based on fields' values
Hi Nik,
Inline below.
On Aug 15, 2010, at 5:01 PM, Nik Kolev wrote:
> Hi,
>
> I am researching the possibility of using Lucene for discovering
> clusters of documents and since I am new to Lucene I decided to
> ask the community for advice before I poke the APIs and the internals.
> Your input will be invaluable!
>
> Here's the use case. Documents arrive from different feeds. Each feed
> produces millions and millions of documents. Documents are structured
> and share certain "interesting" fields. For the purpose of
illustration
> here's a trivial example. Let's say the documents on each feed
represent
> shoes. A shoe has an ID (uniquely identifying it within its feed) and
a
> SIZE [1]. I want to be able to ask Lucene (assuming the shoe documents
> are indexed of course) for all of the clusters of shoes that sport the
same
> size. A cluster of shoes is just the IDs of the shoes that got grouped
> together due to having the same value of the SIZE field.
>
> I don't think that doing this brute force will perform. Here's what I
> mean when I say brute force. I looked at IndexReader, and I saw that
> I can get the distinct values for an indexed field. So assuming that
> each feed will have its own index (I don't think I can get away with
> a single index for all feeds [2]), I can get the union of all distinct
> values for the interesting field across all indexes (one per each
> document feed). Then for each distinct value I can do a MultiSearcher
> search across these indexes getting the IDs of the documents .
>
> My gut tells me that the brute force approach won't perform [3]. And
> here's where you guys come in - is it possible to ask lucene to give
> me the groups of records (across indixes) that share the value for a
> given field? Is there something else in API that I can take advantage
> of and get what I need faster? If I am to extend Lucene to allow for
> this sort of thing, where should I start, what documents should I
> read,...?
This feature, called faceting -- amongst other things, comes out of the
box in Solr (http://lucene.apache.org/solr)
>
> Thanks very much again for your advise,
> -nik
>
> [1] Shoe document layout:
> SHOE
> |--ID
> |--SIZE
>
> [2] The volume of documents I get from each feed is huge. Also, even
> though a lot of fields are shared the format across feeds varies
> and a feed may not have some of the "interesting" fields of
> another feed.
>
> [3] The dataset I am talking about is rich and the brute force
approach
> means that I need to be doing tens of millions of searches just to
> group on one field. Also I most likely will blow my heap up if I
try to
> load all of the values in memory all at once.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem using Solr/Lucene:
http://www.lucidimagination.com/search
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: cluster documents based on fields' values
Posted by Grant Ingersoll <gs...@apache.org>.
Hi Nik,
Inline below.
On Aug 15, 2010, at 5:01 PM, Nik Kolev wrote:
> Hi,
>
> I am researching the possibility of using Lucene for discovering
> clusters of documents and since I am new to Lucene I decided to
> ask the community for advice before I poke the APIs and the internals.
> Your input will be invaluable!
>
> Here's the use case. Documents arrive from different feeds. Each feed
> produces millions and millions of documents. Documents are structured
> and share certain "interesting" fields. For the purpose of illustration
> here's a trivial example. Let's say the documents on each feed represent
> shoes. A shoe has an ID (uniquely identifying it within its feed) and a
> SIZE [1]. I want to be able to ask Lucene (assuming the shoe documents
> are indexed of course) for all of the clusters of shoes that sport the same
> size. A cluster of shoes is just the IDs of the shoes that got grouped
> together due to having the same value of the SIZE field.
>
> I don't think that doing this brute force will perform. Here's what I
> mean when I say brute force. I looked at IndexReader, and I saw that
> I can get the distinct values for an indexed field. So assuming that
> each feed will have its own index (I don't think I can get away with
> a single index for all feeds [2]), I can get the union of all distinct
> values for the interesting field across all indexes (one per each
> document feed). Then for each distinct value I can do a MultiSearcher
> search across these indexes getting the IDs of the documents .
>
> My gut tells me that the brute force approach won't perform [3]. And
> here's where you guys come in - is it possible to ask lucene to give
> me the groups of records (across indixes) that share the value for a
> given field? Is there something else in API that I can take advantage
> of and get what I need faster? If I am to extend Lucene to allow for
> this sort of thing, where should I start, what documents should I
> read,...?
This feature, called faceting -- amongst other things, comes out of the box in Solr (http://lucene.apache.org/solr)
>
> Thanks very much again for your advise,
> -nik
>
> [1] Shoe document layout:
> SHOE
> |--ID
> |--SIZE
>
> [2] The volume of documents I get from each feed is huge. Also, even
> though a lot of fields are shared the format across feeds varies
> and a feed may not have some of the "interesting" fields of
> another feed.
>
> [3] The dataset I am talking about is rich and the brute force approach
> means that I need to be doing tens of millions of searches just to
> group on one field. Also I most likely will blow my heap up if I try to
> load all of the values in memory all at once.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org