Caching IndexSearcher in a webapp [was: Find "latest" document (before a certain date)]

[Per Lindberg <pe...@implior.com>]

> Från: Karl Wettin [mailto:karl.wettin@gmail.com] 

> 29 aug 2007 kl. 12.29 skrev Per Lindberg:
> 
> >> how about using a RangeQuery and pick the hit with the
> >> greatest document number?
> >
> > Yep, that did the trick! There seems to be no Filter that can do
> > the final picking of the highest date, so I had to do that after the
> > search.
> >
> > I use IndexSearcher.search with a RangeFilter,
> > I presume that's just as efficient as a RangeQuery?
> 
> It depends, espescially on how you use reuse the filter.

For each search request (it's a webapp) I currently create
a new IndexSearcher, new Filter and new Sort, call
searcher.search(query, filter, sorter) and later searcher.close().

The literature says that it is desirable to cache the IndexSearcher,
but there's no mention of the memory cost! Since it is said to
take a long time to create, I presume that the IndexSearcher
reads the index files and keeps a lot of stuff in memory, so
the thought of caching one for each HttpSession gives me bad vibes.

(Also keeping open files; the file locking scheme in NTFS
can prevent Tomcat from doing hot redeploy if the webapp
has open files).

> Benchmark to be sure

So far searches with Lucene feel astonishingly fast! Yay! :-)




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

SV: Caching IndexSearcher in a webapp [was: Find "latest" document (before a certain date)]

[Per Lindberg <pe...@implior.com>]

Kalle and Patrick: many thanks for the suggestions!

Caching the IndexSearcher in the ServletContext sounds like a very good idea.
However, I have to index a number of databases, each with a different Lucene
index. So keeping an IndexSearcher for each may come with a prohibitive
memory cost. But as far as I can tell, speed is not a problem; creating a new
IndexSearcher for each new search is outweighed by HTTP protocol latency.

Thanks again!




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Caching IndexSearcher in a webapp [was: Find "latest" document (before a certain date)]

[Karl Wettin <ka...@gmail.com>]

29 aug 2007 kl. 14.32 skrev Per Lindberg:

> For each search request (it's a webapp) I currently create
> a new IndexSearcher, new Filter and new Sort, call
> searcher.search(query, filter, sorter) and later searcher.close().

You really want to reuse the IndexSearcher until new data has
been added to the index. I suppose the same thing goes for filters
and perhaps even sorts?

Start here:

http://wiki.apache.org/lucene-java/ImproveSearchingSpeed


-- 
kalle



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Caching IndexSearcher in a webapp [was: Find "latest" document (before a certain date)]

[Patrick Turcotte <pa...@gmail.com>]

Hi,

Answers in the text.
> For each search request (it's a webapp) I currently create
> a new IndexSearcher, new Filter and new Sort, call
> searcher.search(query, filter, sorter) and later searcher.close().
>
> The literature says that it is desirable to cache the IndexSearcher,
> but there's no mention of the memory cost! Since it is said to
> take a long time to create, I presume that the IndexSearcher
> reads the index files and keeps a lot of stuff in memory, so
> the thought of caching one for each HttpSession gives me bad vibes.

Why don't you put into the context scope
[servletContext.setAttribute("index", IndexSearcher)] ?

You can have it initialized upon startup with init() and cleanup on
shutdown with destroy()

Hope this helps.

Patrick

>
> (Also keeping open files; the file locking scheme in NTFS
> can prevent Tomcat from doing hot redeploy if the webapp
> has open files).
>
> > Benchmark to be sure
>
> So far searches with Lucene feel astonishingly fast! Yay! :-)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org