You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Morus Walter <mo...@tanto-xipolis.de> on 2003/12/18 10:58:08 UTC
Re: best way of reusing IndexSearcher objects
Some time ago (Nov 5) Otis suggested to use one IndexSearcher instance for
search and recreate this instance when the index changed:
> Use a single instance of IndexSearcher.
> When you detect that the index has changed, through that instance (see
> javadoc for the exact method name, I don't recall its exact name now),
> discard that instance, and make a new one.
My problem is that I don't find an apropriate method in the javadoc.
Do I miss something?
There are static methods 'lastModified' in the IndexReader class.
So one might use these (together with a timestamp, when the IndexSearcher
was opened) to do the check.
But it would be nice to have some 'isUpToDate' method in the class itself.
Or should I do that myself by subclassing IndexSearcher?
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best way of reusing IndexSearcher objects
Posted by Doug Cutting <cu...@lucene.com>.
Doug Cutting wrote:
> That's true. If you're doing updates (as opposed to just additions)
> then you probably want to do something like:
> 1. keep a single open IndexReader used by all searches
> 2. Every few minutes, process updates as follows:
> a. open a second IndexReader
> b. delete all documents that will be updated
> c. close this IndexReader, to flush deletions
> d. open an IndexWriter
> e. add all documents that are updated
> f. close the IndexReader
> g. replace the IndexReader used for searches (1, above)
Correction: (f) should read "close the IndexWriter".
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best way of reusing IndexSearcher objects
Posted by Dror Matalon <dr...@zapatec.com>.
On Thu, Dec 18, 2003 at 11:40:32AM -0800, Doug Cutting wrote:
> Dror Matalon wrote:
> >There are two issues:
> >1. Having new searches start using the new index only when it's ready,
> >not in a "half baked" state, which means that you have to synchronize
> >the switch from the old index to the new one.
>
> That's true. If you're doing updates (as opposed to just additions)
> then you probably want to do something like:
At this point, we're just doing additions.
> 1. keep a single open IndexReader used by all searches
> 2. Every few minutes, process updates as follows:
> a. open a second IndexReader
> b. delete all documents that will be updated
> c. close this IndexReader, to flush deletions
> d. open an IndexWriter
> e. add all documents that are updated
> f. close the IndexReader
> g. replace the IndexReader used for searches (1, above)
>
> >2. It's not trivial to figure out when it's safe to discard the old
> >index; all existing searches are done with it.
> >
> >To make things more complicated, the Hits object is dependent on your
> >IndexSearcher object, so if you have Hits objects in use you probably
> >can't close your IndexSearcher.
> >
> >Is this a correct analysis or is there an obvious strategy to work
> >around this issue?
>
> Right, you cannot safely close the IndexReader that's being used for
> searching. Rather, just drop it on the floor and let it get garbage
> collected. Its files will be closed when this happens. Provided you're
> not updating more frequently than the garbage collector runs, you should
> only ever have two IndexReaders open and shouldn't run into file handle
> issues.
Actually, rather than relying on the GC, my plan is to always have two
indexed open, the current one and the previous one, and when I detect
that it's time to switch, I then close the oldest index. Luckily in a
web environment, we know that searches are not going to be around for
several minutes, and I make sure that my indexes don't change more often
than every 15 minutes.
Thanks for the prompt and detailed feedback.
Dror
>
> Doug
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best way of reusing IndexSearcher objects
Posted by Morus Walter <mo...@tanto-xipolis.de>.
Doug Cutting writes:
> Dror Matalon wrote:
> > There are two issues:
> > 1. Having new searches start using the new index only when it's ready,
> > not in a "half baked" state, which means that you have to synchronize
> > the switch from the old index to the new one.
>
> That's true. If you're doing updates (as opposed to just additions)
> then you probably want to do something like:
> 1. keep a single open IndexReader used by all searches
> 2. Every few minutes, process updates as follows:
> a. open a second IndexReader
> b. delete all documents that will be updated
> c. close this IndexReader, to flush deletions
> d. open an IndexWriter
> e. add all documents that are updated
> f. close the IndexReader
> g. replace the IndexReader used for searches (1, above)
>
Right. As long as you can control the reader instance from the update
process, it's better to do so, instead of checking, if the reader for search
is still up to date in the reader itself.
> > 2. It's not trivial to figure out when it's safe to discard the old
> > index; all existing searches are done with it.
> >
> > To make things more complicated, the Hits object is dependent on your
> > IndexSearcher object, so if you have Hits objects in use you probably
> > can't close your IndexSearcher.
> >
> > Is this a correct analysis or is there an obvious strategy to work
> > around this issue?
>
> Right, you cannot safely close the IndexReader that's being used for
> searching. Rather, just drop it on the floor and let it get garbage
> collected. Its files will be closed when this happens. Provided you're
> not updating more frequently than the garbage collector runs, you should
> only ever have two IndexReaders open and shouldn't run into file handle
> issues.
>
I guess the alternative would be to have a reference counting that is
increased whenever a search starts and decreased when the hits object
is no longer used.
You could then set a flag and close the index when the count reaches 0.
Thanks for the comments.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best way of reusing IndexSearcher objects
Posted by Doug Cutting <cu...@lucene.com>.
Dror Matalon wrote:
> There are two issues:
> 1. Having new searches start using the new index only when it's ready,
> not in a "half baked" state, which means that you have to synchronize
> the switch from the old index to the new one.
That's true. If you're doing updates (as opposed to just additions)
then you probably want to do something like:
1. keep a single open IndexReader used by all searches
2. Every few minutes, process updates as follows:
a. open a second IndexReader
b. delete all documents that will be updated
c. close this IndexReader, to flush deletions
d. open an IndexWriter
e. add all documents that are updated
f. close the IndexReader
g. replace the IndexReader used for searches (1, above)
> 2. It's not trivial to figure out when it's safe to discard the old
> index; all existing searches are done with it.
>
> To make things more complicated, the Hits object is dependent on your
> IndexSearcher object, so if you have Hits objects in use you probably
> can't close your IndexSearcher.
>
> Is this a correct analysis or is there an obvious strategy to work
> around this issue?
Right, you cannot safely close the IndexReader that's being used for
searching. Rather, just drop it on the floor and let it get garbage
collected. Its files will be closed when this happens. Provided you're
not updating more frequently than the garbage collector runs, you should
only ever have two IndexReaders open and shouldn't run into file handle
issues.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best way of reusing IndexSearcher objects
Posted by Dror Matalon <dr...@zapatec.com>.
> Some time ago (Nov 5) Otis suggested to use one IndexSearcher instance for
> search and recreate this instance when the index changed:
>
> > Use a single instance of IndexSearcher.
> > When you detect that the index has changed, through that instance (see
> > javadoc for the exact method name, I don't recall its exact name now),
> > discard that instance, and make a new one.
>
Isn't this easier said than done? In a web environment, you have hits
coming in and searching continuously.
There are two issues:
1. Having new searches start using the new index only when it's ready,
not in a "half baked" state, which means that you have to synchronize
the switch from the old index to the new one.
2. It's not trivial to figure out when it's safe to discard the old
index; all existing searches are done with it.
To make things more complicated, the Hits object is dependent on your
IndexSearcher object, so if you have Hits objects in use you probably
can't close your IndexSearcher.
Is this a correct analysis or is there an obvious strategy to work
around this issue?
Regards,
Dror
--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: best way of reusing IndexSearcher objects
Posted by Otis Gospodnetic <ot...@yahoo.com>.
You could subclass IndexSearcher like you said. You could patch
IndexSearcher by adding this method. You could also change the search
method to do this check automatically. You could also add
setAutoRefresh(boolean) method to IndexSearcher and then to automatic
refresh only if this was set to true (default could remain false, for
backward compatibility).
diff -u patcher in Bugzilla are very welcome! :)
Otis
--- Morus Walter <mo...@tanto-xipolis.de> wrote:
> Some time ago (Nov 5) Otis suggested to use one IndexSearcher
> instance for
> search and recreate this instance when the index changed:
>
> > Use a single instance of IndexSearcher.
> > When you detect that the index has changed, through that instance
> (see
> > javadoc for the exact method name, I don't recall its exact name
> now),
> > discard that instance, and make a new one.
>
> My problem is that I don't find an apropriate method in the javadoc.
> Do I miss something?
>
> There are static methods 'lastModified' in the IndexReader class.
> So one might use these (together with a timestamp, when the
> IndexSearcher
> was opened) to do the check.
>
> But it would be nice to have some 'isUpToDate' method in the class
> itself.
> Or should I do that myself by subclassing IndexSearcher?
>
> Morus
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________
Do you Yahoo!?
New Yahoo! Photos - easier uploading and sharing.
http://photos.yahoo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org