optimized reopen?

[Robert Engels <re...@ix.netcom.com>]

Is there any way to optimize the closing & reopening of an Index?

Since the IndexReader.open() opens a MultiReader is there are multiple
segments, it seems a reopen() method could be implemented, which detects
which segments are the same as the current open index, and then passes those
SegementReaders to a new Multireader rather than creating new SegmentReaders
for the unmodified segments.

Does this sounds feasible? Would it improve the performance?

Thanks

RE: optimized reopen?

[Robert Engels <re...@ix.netcom.com>]

I did an implementation of IndexReader.reopen() in my modified Lucene lib.

It is over 200% faster than closing and opening the index.

I just made the semantics of IndexReader.reopen() be that the original
IndexReader is now closed, and can no longer be used (since I cannot close()
the original, otherwise the SegmentReaders will close, so I close the unused
SegmentReaders directly). In order to make this work I also needed to make
commit() public, since the commit() needs to be performed prior to the
segment merging, and I needed to add a package method to MultiReader to get
the underlying subReaders.

Although this may not be useful to many, in our case since we use a single
shared reader and a single exclusive writer in a server process (with
external locking, so we can guarantee than when the IndexReader is "rolled",
there are no references to the previous reader.

Robert

-----Original Message-----
From: Yonik Seeley [mailto:yseeley@gmail.com]
Sent: Wednesday, May 11, 2005 4:50 PM
To: java-dev@lucene.apache.org; rengels@ix.netcom.com
Subject: Re: optimized reopen?


Things are cached using an IndexReader as the key, so you would have
to be careful not to break the current behaviour (that an
IndexReader's view of an index doesn't change - deletes from that
specific reader aside).

Maybe you could invoke reopen() on an existing IndexReader and it
would return a new IndexReader that shares sub-readers that haven't
changed?  But I don't think that sub-readers can be shared like this
right now... the difficulty may lie in deleted-docs.

-Yonik


On 5/11/05, Robert Engels <re...@ix.netcom.com> wrote:
> Is there any way to optimize the closing & reopening of an Index?
>
> Since the IndexReader.open() opens a MultiReader is there are multiple
> segments, it seems a reopen() method could be implemented, which detects
> which segments are the same as the current open index, and then passes
those
> SegementReaders to a new Multireader rather than creating new
SegmentReaders
> for the unmodified segments.
>
> Does this sounds feasible? Would it improve the performance?
>
> Thanks
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

RE: optimized reopen?

[Robert Engels <re...@ix.netcom.com>]

I was pretty sure reopen() would need to return a new IndexReader instance,
because otherwise the 'indices never change' paradigm would break, but it
would seem that I could share the existing SegmentReaders with the new
IndexReader.

Why do you think the SegmentReaders cannot be shared?

R

-----Original Message-----
From: Yonik Seeley [mailto:yseeley@gmail.com]
Sent: Wednesday, May 11, 2005 4:50 PM
To: java-dev@lucene.apache.org; rengels@ix.netcom.com
Subject: Re: optimized reopen?


Things are cached using an IndexReader as the key, so you would have
to be careful not to break the current behaviour (that an
IndexReader's view of an index doesn't change - deletes from that
specific reader aside).

Maybe you could invoke reopen() on an existing IndexReader and it
would return a new IndexReader that shares sub-readers that haven't
changed?  But I don't think that sub-readers can be shared like this
right now... the difficulty may lie in deleted-docs.

-Yonik


On 5/11/05, Robert Engels <re...@ix.netcom.com> wrote:
> Is there any way to optimize the closing & reopening of an Index?
>
> Since the IndexReader.open() opens a MultiReader is there are multiple
> segments, it seems a reopen() method could be implemented, which detects
> which segments are the same as the current open index, and then passes
those
> SegementReaders to a new Multireader rather than creating new
SegmentReaders
> for the unmodified segments.
>
> Does this sounds feasible? Would it improve the performance?
>
> Thanks
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org

Re: optimized reopen?

[Yonik Seeley <ys...@gmail.com>]

Things are cached using an IndexReader as the key, so you would have
to be careful not to break the current behaviour (that an
IndexReader's view of an index doesn't change - deletes from that
specific reader aside).

Maybe you could invoke reopen() on an existing IndexReader and it
would return a new IndexReader that shares sub-readers that haven't
changed?  But I don't think that sub-readers can be shared like this
right now... the difficulty may lie in deleted-docs.

-Yonik


On 5/11/05, Robert Engels <re...@ix.netcom.com> wrote:
> Is there any way to optimize the closing & reopening of an Index?
> 
> Since the IndexReader.open() opens a MultiReader is there are multiple
> segments, it seems a reopen() method could be implemented, which detects
> which segments are the same as the current open index, and then passes those
> SegementReaders to a new Multireader rather than creating new SegmentReaders
> for the unmodified segments.
> 
> Does this sounds feasible? Would it improve the performance?
> 
> Thanks
> 
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org