You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by wgggfiy <wu...@qq.com> on 2013/05/13 05:05:00 UTC
why did I build index slower and slower ？
My situation is that There are 10,000,000 documents, and I Build index every
5,000 documents. while *in every build*, I follow these steps:
IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_40, analyzer);
iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);
iwc.setRAMBufferSizeMB(ram); int maxDocs =
itmParamParser.maxBufferedDocs;
iwc.setMaxBufferedDocs(100000); logger.info("build |
RAMBufferSizeMB="+ram+" MB, MaxBufferedDocs="+maxDocs);
writer = new IndexWriter(directory, iwc);and I found it build slower and
slower, so that I couldn't stand the speed !Any suggestion, thank guys.
-----
--------------------------
Email: wuqiu.main@qq.com
--------------------------
--
View this message in context: http://lucene.472066.n3.nabble.com/why-did-I-build-index-slower-and-slower-tp4062798.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
Re: why did I build index slower and slower ？
Posted by wgggfiy <wu...@qq.com>.
up
-----
--------------------------
Email: wuqiu.main@qq.com
--------------------------
--
View this message in context: http://lucene.472066.n3.nabble.com/why-did-I-build-index-slower-and-slower-tp4062798p4063395.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
Re: why did I build index slower and slower ？
Posted by wgggfiy <wu...@qq.com>.
En, thanke you. I also found the question that I should make the writer a
singleton. and the writer commited and closed every batch. That is,In every
buildIndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_40,
analyzer);iwc.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);iwc.setRAMBufferSizeMB(ram);int
maxDocs =
itmParamParser.maxBufferedDocs;iwc.setMaxBufferedDocs(100000);logger.info("build
| RAMBufferSizeMB="+ram+" MB, MaxBufferedDocs="+maxDocs);writer = new
IndexWriter(directory, iwc);.........writer.addDocument(document,
analyzer);......writer.commit(userData);writer.close();
-----
--------------------------
Email: wuqiu.main@qq.com
--------------------------
--
View this message in context: http://lucene.472066.n3.nabble.com/why-did-I-build-index-slower-and-slower-tp4062798p4062849.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
Re: why did I build index slower and slower ？
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2013-05-13 at 05:05 +0200, wgggfiy wrote:
> My situation is that There are 10,000,000 documents, and I Build index every
> 5,000 documents. while *in every build*, I follow these steps:
> IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_40, analyzer);
You skipped the part where you commit and close. If you are optimizing
the index down to a low number of segments, that would explain most of
the slowdown.
Another part is that you open the index for each batch. That takes a bit
of time. If you have frequent batch runs, you might want to switch to a
setup where the index writer is persistent.
- Toke Eskildsen, state and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org