You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Patrick Ng <pa...@jobsdb.com> on 2008/02/17 18:44:36 UTC
Index Search becomes very slow during index update
Hi,
Our company has been using Lucene.Net. Our index file size is small
(<200MB), the number of document is small (<100000), and our searching
is very fast (below 200ms). And we also have a service which updates
(i.e. delete and add) documents in the index once every minute.
What we found out is that whenever there is an update (frequency is 4-7
updates per second, come in burst of duration of 5-10sec once a minute),
the searching time will in many cases increase to >1000ms. Sometimes it
could go up to 5000ms! We have already looked through our code but
couldn't find any obvious locking problem. Please note that we're also
optimizing our index once every 10 minutes.
Would really appreciate if anyone could give us some tips on what we
should look at.
Thanks,
- Patrick
RE: Index Search becomes very slow during index update
Posted by Nic Wise <Ni...@bbc.com>.
Try tweaking the merge factor etc:
_Writer.SetUseCompoundFile(true); // even though is default.
_Writer.minMergeDocs = _WriterOptions.MinMergeDocuments;
_Writer.maxMergeDocs = _WriterOptions.MaxMergeDocuments;
_Writer.SetMergeFactor(_WriterOptions.MergeFactor);
We had min set to (I think) 10 or so, Max was MaxInt, which was BAD, and
we changed it to around 100K - I guess in your case make it around 10K?
I can't remember what we had for the merge factor.
-----Original Message-----
From: Patrick Ng [mailto:patrickng@jobsdb.com]
Sent: 17 February 2008 17:45
To: lucene-net-user@incubator.apache.org
Cc: Peter Chiu; Wing Fong
Subject: Index Search becomes very slow during index update
Hi,
Our company has been using Lucene.Net. Our index file size is small
(<200MB), the number of document is small (<100000), and our searching
is very fast (below 200ms). And we also have a service which updates
(i.e. delete and add) documents in the index once every minute.
What we found out is that whenever there is an update (frequency is 4-7
updates per second, come in burst of duration of 5-10sec once a minute),
the searching time will in many cases increase to >1000ms. Sometimes it
could go up to 5000ms! We have already looked through our code but
couldn't find any obvious locking problem. Please note that we're also
optimizing our index once every 10 minutes.
Would really appreciate if anyone could give us some tips on what we
should look at.
Thanks,
- Patrick
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this
This e-mail has been sent by one of the following wholly-owned subsidiaries of the BBC:
BBC Worldwide, Registration Number: 1420028 England, Registered Address: Woodlands, 80 Wood Lane, London W12 0TT
BBC World, Registration Number: 04514407 England, Registered Address: Woodlands, 80 Wood Lane, London W12 0TT
BBC World Distribution Limited, Registration Number: 04514408, Registered Address: Woodlands, 80 Wood Lane, London W12 0TT
RE: How do I update a document without first delete and then add
Posted by Michael Garski <mg...@myspace.com>.
Min,
The only way to update a document is to first delete the existing one,
then re-add it. There is no way to retain the same document id when
this is done.
Michael
-----Original Message-----
From: Min Yin [mailto:yin@AI.SRI.COM]
Sent: Tuesday, June 10, 2008 4:54 PM
To: lucene-net-user@incubator.apache.org
Subject: How do I update a document without first delete and then add
Hi there,
Sorry for the naive question, but I couldn't seem to find a way to
update a document (i.e., one or multiple fields of a document) without
first delete the old one and then add a new one, in other words, I'd
like to keep the old document id. But I only find
IndexWriter.updateDocument(term, doc) which "Updates a document by first
deleting the document(s) containing |term| and then adding the new
document". Any idea?
Thanks in Advance!
Min
How do I update a document without first delete and then add
Posted by Min Yin <yi...@AI.SRI.COM>.
Hi there,
Sorry for the naive question, but I couldn't seem to find a way to
update a document (i.e., one or multiple fields of a document) without
first delete the old one and then add a new one, in other words, I'd
like to keep the old document id. But I only find
IndexWriter.updateDocument(term, doc) which "Updates a document by first
deleting the document(s) containing |term| and then adding the new
document". Any idea?
Thanks in Advance!
Min