large indexes

[Scott Smith <ss...@mainstreamdata.com>]

I have the need to create an index which will potentially have a
million+ documents.  I know Lucene can accomplish this.  However, the
other requirement is that I need to be continually updating it during
the date (adding 1-30 documents/minute).  I guess I had thought that I
might try to have an active index that I keep today's documents in and
then a large cumulative index of all of the previous day's documents.
At the end of each day, I would create a new version of the cumulative
index adding in that day's documents.  I would use the multi-index
search to search the current day and cumulative indexes.

 

Has anyone dealt with this design issue?  Is this a reasonable approach?
What have other people done?

 

Scott 

 

 

Re: large indexes

[Doug Cutting <cu...@apache.org>]

Scott Smith wrote:
> I have the need to create an index which will potentially have a
> million+ documents.  I know Lucene can accomplish this.  However, the
> other requirement is that I need to be continually updating it during
> the date (adding 1-30 documents/minute).

Have a look at this thread:

http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg12709.html

Technorati updates hundreds of blog entries per minute with a single 
Lucene index.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org