You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Chris David <ch...@snapstream.com> on 2006/08/29 02:13:32 UTC
Lucene.Net Indexing Large Databases
Hello Everyone,
I have been trying to get Lucene.Net to index a large number of
documents but I have been having trouble when batch processing around
800,000 documents at a time. At first I tried indexing using a
FSDirectory as my indexing directory, but that process takes incredibly
long. I also tried using a Ram Directory as my indexing directory and
then merging with a FSDirectory after every 10,000 new documents.
Using this process would take a while and Take about 1.8 GB of Ram and
eventually crash with an out of memory error. Right now each document
isn't more than a few words each and the batch processing is using up a
huge chunk of the system resources.
I would appreciate some help with batch processing large numbers of
documents at a time. In batch processing should I batch processing
using a Ram Directory and then merge indexes with an FSDirectory? Does
Lucene indexing usually take large amounts of Ram at a time? Can
Lucene.Net handle batch processing millions of documents at a time?
Thanks,
Chris
Snapstream Media
RE: Lucene.Net Indexing Large Databases
Posted by "Kauler, Leto S" <le...@education.tas.gov.au>.
> I have been trying to get Lucene.Net to index a large number of
documents but I have
> been having trouble when batch processing around 800,000 documents at
a time. At
> first I tried
Have you checked out IndexWriter's MergeFactor and Min/MaxMergeDocs? We
only index up to a couple of hundred thousands docs but we use those
properties to manage RAM-usage versus disk-usage, ie use more RAM to
give less frequent disk writing.
This page is good (or google "lucene mergefactor"):
http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html
--Leto
CONFIDENTIALITY NOTICE AND DISCLAIMER
Information in this transmission is intended only for the person(s) to whom it is addressed and may contain privileged and/or confidential information. If you are not the intended recipient, any disclosure, copying or dissemination of the information is unauthorised and you should delete/destroy all copies and notify the sender. No liability is accepted for any unauthorised use of the information contained in this transmission.
This disclaimer has been automatically added.