You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Laxmi Lal Menaria <lm...@chambal.com> on 2007/01/18 10:51:13 UTC
Increase optimize speed
Hello,
I have created a sample application using lucene, and indexed 5-6
text files data. I am Indexing the files data by reading line by
line, and optimize the index after every file. my text file size
is appx 50 MB. after 3-4 files optimize takes too much time, appx 10-15
minutes, I am doing the current logic right now... pls suggest if I
wrong? or how to increase my optimize speed..
My Platform is : Windows XP SP2, .Net Framework2.0, and LuceneNet2.0, VB.Net 2005
Dim lucWriter As IndexWriter = New IndexWriter"C:\LM", New StandardAnalyzer, True)
lucWriter.SetMergeFactor(1000)
lucWriter.SetMaxMergeDocs(2147483647)
lucWriter.SetMaxBufferedDocs(
1000
)
Dim SW As StreamReader = New StreamReader(_sFileName)
sFieldvalue =
SW
.ReadToEnd
Dim lDoc As Lucene.Net.Documents.Document = New Lucene.Net.Documents.Document
Dim lFieldName As Lucene.Net.Documents.Field
lFieldName = New Lucene.Net.Documents.Field("filename", _sFileName,
Lucene.Net.Documents.Field.Store.YES,
Lucene.Net.Documents.Field.Index.TOKENIZED)
lDoc.Add(lFieldName)
lFieldName = Nothing
lFieldName = New Lucene.Net.Documents.Field("filevalue", sFieldValue,
Lucene.Net.Documents.Field.Store.YES,
Lucene.Net.Documents.Field.Index.TOKENIZED)
lDoc.Add(lFieldName)
lFieldName = Nothing
lucWriter
.AddDocument(lDoc)
lucWriter.Optimize()
lucWriter.Close()
Thanks,
LMenaria
RE: Increase optimize speed
Posted by George Aroush <ge...@aroush.net>.
Hi,
Optimizing each time you add a new file is the wrong thing to do. Once a
night or a weekend should be enough. However, you need to experiment to
find out how often you need to optimize; it all depends on how much change
your index is going through and how frequently.
One thing will speed up index optimization is fester hard-drive. A 10,000
RPM SCSI hard-drive will give you a big speed up over a generic 54,000 RPM
EIDE hard-drive. Even better a second hard-drive, specially if you are
merging indexes, will give you even better boost if you merge from one
hard-drive to the other.
Regards,
-- George Aroush
-----Original Message-----
From: Michael Garski [mailto:mgarski@mac.com]
Sent: Thursday, January 18, 2007 1:01 PM
To: lucene-net-user@incubator.apache.org
Subject: Re: Increase optimize speed
Laxmi,
An optimization will rewrite the entire index by merging all segment files
into one file, which can take some time if you index occupies a lot of space
on disk. Optimizing after each file is not necessary, and is best done when
you are done indexing or on a periodic basis if you are continually adding
or deleting documents.
I have indexes in the 2-4 GB range that can take up to an hour to optimize,
which is performed on a nightly basis.
Michael
Laxmi Lal Menaria wrote:
> Hello,
>
> I have created a sample application using lucene, and indexed 5-6
> text files data. I am Indexing the files data by reading line by line,
> and optimize the index after every file. my text file size is appx 50
> MB. after 3-4 files optimize takes too much time, appx 10-15 minutes,
> I am doing the current logic right now... pls suggest if I wrong? or
> how to increase my optimize speed..
>
> My Platform is : Windows XP SP2, .Net Framework2.0, and LuceneNet2.0,
> VB.Net 2005
>
> Dim lucWriter As IndexWriter = New IndexWriter"C:\LM", New
> StandardAnalyzer, True)
> lucWriter.SetMergeFactor(1000)
>
> lucWriter.SetMaxMergeDocs(2147483647)
>
> lucWriter.SetMaxBufferedDocs(
> 1000
> )
>
> Dim SW As StreamReader = New StreamReader(_sFileName)
>
> sFieldvalue =
> SW
> .ReadToEnd
>
> Dim lDoc As Lucene.Net.Documents.Document = New
> Lucene.Net.Documents.Document
>
> Dim lFieldName As Lucene.Net.Documents.Field
>
> lFieldName = New Lucene.Net.Documents.Field("filename", _sFileName,
> Lucene.Net.Documents.Field.Store.YES,
> Lucene.Net.Documents.Field.Index.TOKENIZED)
>
> lDoc.Add(lFieldName)
>
> lFieldName = Nothing
>
> lFieldName = New Lucene.Net.Documents.Field("filevalue", sFieldValue,
> Lucene.Net.Documents.Field.Store.YES,
> Lucene.Net.Documents.Field.Index.TOKENIZED)
>
> lDoc.Add(lFieldName)
>
> lFieldName = Nothing
>
> lucWriter
> .AddDocument(lDoc)
>
> lucWriter.Optimize()
>
> lucWriter.Close()
>
> Thanks,
> LMenaria
>
>
>
RE: Increase optimize speed
Posted by mi...@justanswer.com.
I took the approach where I have two indexes. One live, one for
rebuilding/optimizing. Once the second one is ready, the internal singleton
service switches over to the new one (allowing current ones to finish up).
This seems to be seamless to the user's experience.
-----Original Message-----
From: Michael Garski [mailto:mgarski@mac.com]
Sent: Thursday, January 18, 2007 10:01 AM
To: lucene-net-user@incubator.apache.org
Subject: Re: Increase optimize speed
Laxmi,
An optimization will rewrite the entire index by merging all segment files
into one file, which can take some time if you index occupies a lot of space
on disk. Optimizing after each file is not necessary, and is best done when
you are done indexing or on a periodic basis if you are continually adding
or deleting documents.
I have indexes in the 2-4 GB range that can take up to an hour to optimize,
which is performed on a nightly basis.
Michael
Laxmi Lal Menaria wrote:
> Hello,
>
> I have created a sample application using lucene, and indexed 5-6
> text files data. I am Indexing the files data by reading line by line,
> and optimize the index after every file. my text file size is appx 50
> MB. after 3-4 files optimize takes too much time, appx 10-15 minutes,
> I am doing the current logic right now... pls suggest if I wrong? or
> how to increase my optimize speed..
>
> My Platform is : Windows XP SP2, .Net Framework2.0, and LuceneNet2.0,
> VB.Net 2005
>
> Dim lucWriter As IndexWriter = New IndexWriter"C:\LM", New
> StandardAnalyzer, True)
> lucWriter.SetMergeFactor(1000)
>
> lucWriter.SetMaxMergeDocs(2147483647)
>
> lucWriter.SetMaxBufferedDocs(
> 1000
> )
>
> Dim SW As StreamReader = New StreamReader(_sFileName)
>
> sFieldvalue =
> SW
> .ReadToEnd
>
> Dim lDoc As Lucene.Net.Documents.Document = New
> Lucene.Net.Documents.Document
>
> Dim lFieldName As Lucene.Net.Documents.Field
>
> lFieldName = New Lucene.Net.Documents.Field("filename", _sFileName,
> Lucene.Net.Documents.Field.Store.YES,
> Lucene.Net.Documents.Field.Index.TOKENIZED)
>
> lDoc.Add(lFieldName)
>
> lFieldName = Nothing
>
> lFieldName = New Lucene.Net.Documents.Field("filevalue", sFieldValue,
> Lucene.Net.Documents.Field.Store.YES,
> Lucene.Net.Documents.Field.Index.TOKENIZED)
>
> lDoc.Add(lFieldName)
>
> lFieldName = Nothing
>
> lucWriter
> .AddDocument(lDoc)
>
> lucWriter.Optimize()
>
> lucWriter.Close()
>
> Thanks,
> LMenaria
>
>
>
Re: Increase optimize speed
Posted by Michael Garski <mg...@mac.com>.
Laxmi,
An optimization will rewrite the entire index by merging all segment
files into one file, which can take some time if you index occupies a
lot of space on disk. Optimizing after each file is not necessary, and
is best done when you are done indexing or on a periodic basis if you
are continually adding or deleting documents.
I have indexes in the 2-4 GB range that can take up to an hour to
optimize, which is performed on a nightly basis.
Michael
Laxmi Lal Menaria wrote:
> Hello,
>
> I have created a sample application using lucene, and indexed 5-6
> text files data. I am Indexing the files data by reading line by
> line, and optimize the index after every file. my text file size
> is appx 50 MB. after 3-4 files optimize takes too much time, appx 10-15
> minutes, I am doing the current logic right now... pls suggest if I
> wrong? or how to increase my optimize speed..
>
> My Platform is : Windows XP SP2, .Net Framework2.0, and LuceneNet2.0, VB.Net 2005
>
> Dim lucWriter As IndexWriter = New IndexWriter"C:\LM", New StandardAnalyzer, True)
> lucWriter.SetMergeFactor(1000)
>
> lucWriter.SetMaxMergeDocs(2147483647)
>
> lucWriter.SetMaxBufferedDocs(
> 1000
> )
>
> Dim SW As StreamReader = New StreamReader(_sFileName)
>
> sFieldvalue =
> SW
> .ReadToEnd
>
> Dim lDoc As Lucene.Net.Documents.Document = New Lucene.Net.Documents.Document
>
> Dim lFieldName As Lucene.Net.Documents.Field
>
> lFieldName = New Lucene.Net.Documents.Field("filename", _sFileName,
> Lucene.Net.Documents.Field.Store.YES,
> Lucene.Net.Documents.Field.Index.TOKENIZED)
>
> lDoc.Add(lFieldName)
>
> lFieldName = Nothing
>
> lFieldName = New Lucene.Net.Documents.Field("filevalue", sFieldValue,
> Lucene.Net.Documents.Field.Store.YES,
> Lucene.Net.Documents.Field.Index.TOKENIZED)
>
> lDoc.Add(lFieldName)
>
> lFieldName = Nothing
>
> lucWriter
> .AddDocument(lDoc)
>
> lucWriter.Optimize()
>
> lucWriter.Close()
>
> Thanks,
> LMenaria
>
>
>