You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael McCandless <lu...@mikemccandless.com> on 2007/12/10 10:28:59 UTC
Fwd: Can changes on an index be visible to an open IndexSearcher without reopening it?
Carrying this excellent question over to java-dev (see below).
The idea of "incrementally fixing up the FieldCache" has been
discussed before, eg most recently here:
http://www.gossamer-threads.com/lists/lucene/java-dev/53852#53852
And I think this issue from Hoss is working towards it:
https://issues.apache.org/jira/browse/LUCENE-831
I think distributing the caching down to each searcher makes sense?
It's more amenable to incremental updates than a single massive array
at the top with all.
Then, we also talked about faster ways to store a "column-stride
stored field" (eg, norms), here:
http://www.gossamer-threads.com/lists/lucene/java-dev/53878#53878
which would make populating cache of the value for every doc for a
given stored field far faster.
I haven't really looked closely at these (I've been focusing on the
indexing side of the house so far!), but, I do think these ideas are
important to pursue soon (after 2.3). We do really need reopen at the
IndexSearcher level to be as fast as it can be.
Is anyone actively working on this?
Mike
Begin forwarded message:
> From: "Scott Tiger" <m....@gmail.com>
> Date: December 9, 2007 11:23:27 PM EST
> To: java-user@lucene.apache.org
> Subject: Re: Can changes on an index be visible to an open
> IndexSearcher without reopening it?
> Reply-To: java-user@lucene.apache.org
>
> No plan for IndexSearcher.reopen?
> I don't know about cost of creating IndexSearcher instance.
> But we almost need IndexSearcher.reopen instead of IndexReader.reopen.
>
>
> ----
> IndexReader reader = IndexReader.open(...);
> IndexSearcher searcher = new IndexSearcher(reader);
> searcher.search(...);
>
> ...for reopen
>
> IndexReader newReader = reader.reopen();
> if (newReader != reader) {
> searcher.close();
> reader.close();
> reader = newReader;
> newReader = null;
> searcher = new IndexSearcher(reader);
> } else {
> newReader = null;
> }
> searcher.search(...);
>
> ...want simply...
>
> IndexSearcher newSearcher = searcher.reopen();
> if (newSearcher != searcher) {
> searcher.close();
> searcher = newSearcher;
> newSearcher = null;
> }
>
> 2007/11/5, Michael McCandless <lu...@mikemccandless.com>:
>>
>> Unfortunately, no. Once open, the IndexReader/IndexSearcher searches
>> a frozen "point in time" snapshot of the index as it existed when it
>> was first opened.
>>
>> You'll have to open a new searcher in order to see the changes.
>>
>> However, there is work underway now to add a "reopen" method to
>> IndexReader that somewhat lowers the cost of opening a reader (not
>> yet
>> clear by how much). This should be part of the next release (2.3).
>> See here for details:
>>
>> https://issues.apache.org/jira/browse/LUCENE-743
>>
>> Mike
>>
>> "Enrique Lamas" <en...@corp.ya.com> wrote:
>>> Hi,
>>> I have an application using Lucene 2.2.0 that opens an
>>> IndexSearcher only
>>> once to optimize performance, because opening the index is a heavy
>>> operation. My question is, if I modify the index with an
>>> IndexWriter or
>>> IndexModifier, is there any way for the changes to be visible to the
>>> opened IndexSearcher without reopening it?
>>>
>>> Thanks
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Can changes on an index be visible to an open IndexSearcher without reopening it?
Posted by Scott Tiger <m....@gmail.com>.
OK, I'll wait until they are supported in future release. (2.4?)
2007/12/10, Michael McCandless <lu...@mikemccandless.com>:
>
> OK, excellent. I just wanted to make sure this thread is still
> "alive" :) This is an important optimization to decrease cost of
> opening & re-opening searchers.
>
> Mike
>
> Michael Busch wrote:
>
> > Michael McCandless wrote:
> >>
> >> I haven't really looked closely at these (I've been focusing on the
> >> indexing side of the house so far!), but, I do think these ideas are
> >> important to pursue soon (after 2.3). We do really need reopen at
> >> the
> >> IndexSearcher level to be as fast as it can be.
> >>
> >> Is anyone actively working on this?
> >>
> >> Mike
> >>
> >
> > Hi Mike,
> >
> > I haven't worked on the column-stored fields (aka per-doc
> > payloads ;) )
> > yet, but I was planning to start after 2.3 is out. In Atlanta on the
> > ApacheCon, we had a search BOF and people seemed to be really excited
> > about this feature, especially if those fields will be updateable. Of
> > course then it makes total sense to change the FieldCache to load the
> > data from those new fields and also enable it to only reload the
> > values
> > from the changed segments during re-open.
> >
> > So yeah, I was planning to start on these patches soon, and of course
> > any help is more than welcome! :-) I think we should start to make a
> > good design soon, then we can decide who is going to implement what
> > and
> > hopefully get some of the new features in 2.4.
> >
> > -Michael
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Can changes on an index be visible to an open IndexSearcher without reopening it?
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK, excellent. I just wanted to make sure this thread is still
"alive" :) This is an important optimization to decrease cost of
opening & re-opening searchers.
Mike
Michael Busch wrote:
> Michael McCandless wrote:
>>
>> I haven't really looked closely at these (I've been focusing on the
>> indexing side of the house so far!), but, I do think these ideas are
>> important to pursue soon (after 2.3). We do really need reopen at
>> the
>> IndexSearcher level to be as fast as it can be.
>>
>> Is anyone actively working on this?
>>
>> Mike
>>
>
> Hi Mike,
>
> I haven't worked on the column-stored fields (aka per-doc
> payloads ;) )
> yet, but I was planning to start after 2.3 is out. In Atlanta on the
> ApacheCon, we had a search BOF and people seemed to be really excited
> about this feature, especially if those fields will be updateable. Of
> course then it makes total sense to change the FieldCache to load the
> data from those new fields and also enable it to only reload the
> values
> from the changed segments during re-open.
>
> So yeah, I was planning to start on these patches soon, and of course
> any help is more than welcome! :-) I think we should start to make a
> good design soon, then we can decide who is going to implement what
> and
> hopefully get some of the new features in 2.4.
>
> -Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Fwd: Can changes on an index be visible to an open IndexSearcher
without reopening it?
Posted by Michael Busch <bu...@gmail.com>.
Michael McCandless wrote:
>
> I haven't really looked closely at these (I've been focusing on the
> indexing side of the house so far!), but, I do think these ideas are
> important to pursue soon (after 2.3). We do really need reopen at the
> IndexSearcher level to be as fast as it can be.
>
> Is anyone actively working on this?
>
> Mike
>
Hi Mike,
I haven't worked on the column-stored fields (aka per-doc payloads ;) )
yet, but I was planning to start after 2.3 is out. In Atlanta on the
ApacheCon, we had a search BOF and people seemed to be really excited
about this feature, especially if those fields will be updateable. Of
course then it makes total sense to change the FieldCache to load the
data from those new fields and also enable it to only reload the values
from the changed segments during re-open.
So yeah, I was planning to start on these patches soon, and of course
any help is more than welcome! :-) I think we should start to make a
good design soon, then we can decide who is going to implement what and
hopefully get some of the new features in 2.4.
-Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org