Re: caching an indexreader

[Jason Rutherglen <ja...@gmail.com>]

On the topic of RAM consumption, it seems like field caches
could return estimated RAM usage (given they're arrays of
standard Java types)? There's methods of calculating per
platform (I believe relatively accurately).

On Fri, Jun 19, 2009 at 12:11 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Fri, Jun 19, 2009 at 2:40 PM, Scott Smith<ss...@mainstreamdata.com>
> wrote:
> > In my environment, one of the concerns is that new documents are
> > constantly being added (and some documents may be deleted).  This means
> > that when a user does a search and pages through results, it is possible
> > that there are new items coming in which affect the search-thus changing
> > where items are in relation to the pages displayed to the user.
>
> This is a very real problem that most search apps simply ignore.
>
> > It seems to me that one solution would be to cache the Searcher for the
> > duration of the user's search session so that the user's view of the
> > available documents doesn't change.  As I understand it, the user won't
> > see any changes to the index until a new Searcher is created.  However,
> > I'm very sensitive to the amount of session context memory that caching
> > the searcher might take up.  How much memory will caching the searcher
> > cost?  Are there other tradeoff's I need to consider?
>
> Caching the reader & reader affinity (re-using the same reader when
> user pages) is the right approach.
>
> Prior to 2.9 (not yet released) the RAM requirements of multiple
> readers were high, if you sort by field or use the FieldCache
> directly, because a new FieldCache entry is created for the entire
> index, per reader.  It was also slow to reopen for this reason (on a
> large index).
>
> With 2.9, the FieldCache entries are stored per segment, so that a
> newly opened reader only adds entries for new segments in the index.
> Often this is a tiny amount of added RAM, but if a big merge has just
> finished it could be alot of RAM.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

RE: caching an indexreader

[Scott Smith <ss...@mainstreamdata.com>]

Thanks for the comments.  Sounds like I will probably be ok.

-----Original Message-----
From: Jason Rutherglen [mailto:jason.rutherglen@gmail.com] 
Sent: Friday, June 19, 2009 1:50 PM
To: java-user@lucene.apache.org; java-dev@lucene.apache.org
Subject: Re: caching an indexreader

On the topic of RAM consumption, it seems like field caches
could return estimated RAM usage (given they're arrays of
standard Java types)? There's methods of calculating per
platform (I believe relatively accurately).

On Fri, Jun 19, 2009 at 12:11 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Fri, Jun 19, 2009 at 2:40 PM, Scott
Smith<ss...@mainstreamdata.com>
> wrote:
> > In my environment, one of the concerns is that new documents are
> > constantly being added (and some documents may be deleted).  This
means
> > that when a user does a search and pages through results, it is
possible
> > that there are new items coming in which affect the search-thus
changing
> > where items are in relation to the pages displayed to the user.
>
> This is a very real problem that most search apps simply ignore.
>
> > It seems to me that one solution would be to cache the Searcher for
the
> > duration of the user's search session so that the user's view of the
> > available documents doesn't change.  As I understand it, the user
won't
> > see any changes to the index until a new Searcher is created.
However,
> > I'm very sensitive to the amount of session context memory that
caching
> > the searcher might take up.  How much memory will caching the
searcher
> > cost?  Are there other tradeoff's I need to consider?
>
> Caching the reader & reader affinity (re-using the same reader when
> user pages) is the right approach.
>
> Prior to 2.9 (not yet released) the RAM requirements of multiple
> readers were high, if you sort by field or use the FieldCache
> directly, because a new FieldCache entry is created for the entire
> index, per reader.  It was also slow to reopen for this reason (on a
> large index).
>
> With 2.9, the FieldCache entries are stored per segment, so that a
> newly opened reader only adds entries for new segments in the index.
> Often this is a tiny amount of added RAM, but if a big merge has just
> finished it could be alot of RAM.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org