search caching

[Chris Fraschetti <fr...@gmail.com>]

I've got an application that performs millions of searches against a
lucene index, can someone  give me a bit of insight as to the memory
consumption of these searches? Is there a cap on how many are kept
around? is there any way I can disable caching for this type of
search?

-- 
___________________________________________________
Chris Fraschetti
e fraschetti@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: search caching

[David Spencer <da...@tropo.com>]

Chris Fraschetti wrote:

> I've got an application that performs millions of searches against a

If the results are not, say, "personalized", than I suggest some kind of 
web container cache - I use and like OSCache - and it can even cache 
page fragments.

http://www.opensymphony.com/oscache/

> lucene index, can someone  give me a bit of insight as to the memory
> consumption of these searches? Is there a cap on how many are kept
> around? is there any way I can disable caching for this type of
> search?
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: search caching

[Chris Hostetter <ho...@fucit.org>]

: Well, Doug provided this formula a few times, and I don't recall it off
: the top of my head, but I do know that the .tii index file will be
: fully read into memory, so look at the size of that file to get an
: initial idea.

the other big piece of caching that goes on is the FieldCache which is
used for sorting -- if you only ever sort by Score or doc ID, then it's a
non-issue, but if your millions of Searchers are going to involve sorting
on lots of different fields, the complete list of indexed values for each
field used in sorting will be cached in memory per Searcher (close the
Searcher, the WeakRef for the FieldCache is freed up)

I can imagine some scenerios in which it might make sense to do all of the
searches that sort on field A first, then close the searcher and open a
new one and do all the searchers that sort on field B, then close the
searcher, open a new one and do all the searches that sort on field C,
etc....


: > I've got an application that performs millions of searches against a
: > lucene index, can someone  give me a bit of insight as to the memory
: > consumption of these searches? Is there a cap on how many are kept
: > around? is there any way I can disable caching for this type of
: > search?
: >
: > --
: > ___________________________________________________
: > Chris Fraschetti
: > e fraschetti@gmail.com
: >
: > ---------------------------------------------------------------------
: > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: > For additional commands, e-mail: java-user-help@lucene.apache.org
: >
: >
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: search caching

[Otis Gospodnetic <ot...@yahoo.com>]

Searches are not cached (other than possibly by the OS/File system). 
IndexSearcher can/should be cached (by your application), and how much
memory it consumes depends on ... hm, I thought we included a formula
for that in Lucene in Action, but it looks like we only included the
formula for calculating the number of open files during indexing.

Well, Doug provided this formula a few times, and I don't recall it off
the top of my head, but I do know that the .tii index file will be
fully read into memory, so look at the size of that file to get an
initial idea.

Otis

--- Chris Fraschetti <fr...@gmail.com> wrote:

> I've got an application that performs millions of searches against a
> lucene index, can someone  give me a bit of insight as to the memory
> consumption of these searches? Is there a cap on how many are kept
> around? is there any way I can disable caching for this type of
> search?
> 
> -- 
> ___________________________________________________
> Chris Fraschetti
> e fraschetti@gmail.com
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org