You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Chris David <ch...@snapstream.com> on 2006/08/29 02:13:32 UTC

Lucene.Net Indexing Large Databases

Hello Everyone,

 

I have been trying to get Lucene.Net to index a large number of
documents but I have been having trouble when batch processing around
800,000 documents at a time.  At first I tried indexing using a
FSDirectory as my indexing directory, but that process takes incredibly
long.  I also tried using a Ram Directory as my indexing directory and
then merging with a FSDirectory after every 10,000 new documents.
Using this process would take a while and Take about 1.8 GB of Ram and
eventually crash with an out of memory error.  Right now each document
isn't more than a few words each and the batch processing is using up a
huge chunk of the system resources.

 

I would appreciate some help with batch processing large numbers of
documents at a time.  In batch processing should I batch processing
using a Ram Directory and then merge indexes with an FSDirectory?  Does
Lucene indexing usually take large amounts of Ram at a time?  Can
Lucene.Net handle batch processing millions of documents at a time?

 

Thanks,

Chris

Snapstream Media

 


RE: Lucene.Net Indexing Large Databases

Posted by "Kauler, Leto S" <le...@education.tas.gov.au>.
> I have been trying to get Lucene.Net to index a large number of
documents but I have
> been having trouble when batch processing around 800,000 documents at
a time.  At
> first I tried 

Have you checked out IndexWriter's MergeFactor and Min/MaxMergeDocs?  We
only index up to a couple of hundred thousands docs but we use those
properties to manage RAM-usage versus disk-usage, ie use more RAM to
give less frequent disk writing.

This page is good (or google "lucene mergefactor"):
http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html

--Leto


CONFIDENTIALITY NOTICE AND DISCLAIMER

Information in this transmission is intended only for the person(s) to whom it is addressed and may contain privileged and/or confidential information. If you are not the intended recipient, any disclosure, copying or dissemination of the information is unauthorised and you should delete/destroy all copies and notify the sender. No liability is accepted for any unauthorised use of the information contained in this transmission.

This disclaimer has been automatically added.