You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Nikolay Zamosenchuk <ni...@exoplatform.com> on 2010/05/27 11:00:21 UTC

IndexReader.getSequentialSubReaders() usage in Lucene 2.9+

Hi, Dear colleagues!
I have one question concerning IndexReader.getSequentialSubReaders()
and it's usage.

Imagine there is a class extending DirectoryReader or MultiReader.
Usually directory- or multi-reader consists of sub-readers (i.e.
segment-readers). Is it safe enough to return always null in
IndexReader.getSequentialSubReaders() ?

We have a special Query and Weight classes that relies on custom
extended IndexReader. If this method (getSequentialSubReaders) is not
overridden, then our Weight class receives ReadOnlySegmentReader
instance.
I found that IndexSearcher.search(...) method changed in lucene
2.4.1->2.9.2, so subReaders are passed instead of original reader.
I've also found, that returning null in
IndexReader.getSequentialSubReaders() will avoid usage of subReaders.
Seems returning the null helped and tests pessed.
But there is a question, is it safe enough to return null in
IndexReader.getSequentialSubReaders() ? Would it break some internal
logic?

Thanks much!
--
Best regards, Nikolay Zamosenchuk
eXo (UA)

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Using field cache on real time index

Posted by Michael McCandless <lu...@mikemccandless.com>.
When you use .reopen() from an existing reader, the new reader will
share sub-readers with the prior one, and the FieldCache will also
share cache entries for those sub readers, with the prior one.

Still, remember that the incrementality you'll see is in proportion to
the size of the new segments.  EG, if you optimize from the writer
before closing, then IR.reopen buys you nothing since no segments can
be reused from the prior reader.  Also, if a large merge completes,
that large segment must be opened for the new reader.  If this becomes
a problem you should warm the new reader before swapping it into
production.

NRT (IndexWriter.getReader), at least in the current impl, is just
like reopen, except it saves having to close/commit from IndexWriter
(which can be costly since it must fsync all newly created files),
it's also able in some cases to carry pending deletions in RAM instead
of going through the filesystem, and it's able to warm a newly merged
segment in the BG before it's committed to the index, without blocking
the reopen.

Mike

On Mon, Jun 7, 2010 at 2:02 PM, Woolf, Ross <Ro...@bmc.com> wrote:
> I'm looking for some clarification on the use of field cache in a real time index situation.
>
> We are using Lucene in a real time fashion, but we update our reader via IndexReader.reopen() rather than using the IndexWriter.getReader(); After opening a new reader the old reader is closed.
>
> In the book Lucene in Action (2nd) it mentions something regarding since 2.9.2 you can pass in a subreader instead of the reader if you are reopening a reader so that you only have to load the new segment.  But it is not clear on how all of this works.
>
> My big question is does doing it this way mean that I can reuse the portion of the cache that was created with prior readers?  Or does the whole cache have to be rebuilt each time I do a reopen?  I guess what I am asking is if in a real time situation, since I will be reopening readers regularly, is my cache constantly having to be rebuilt from scratch, or can I use this subreader approach to sequentially add to the cache?
>
> Also if the approach lets me sequentially add to the cache, how exactly do I go about dealing with the subreaders?  All I have used before is my primary reader and as I perform a reopen, how do I know what subreader to pass into the fieldcache?
>
> Any info appreciated.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Using field cache on real time index

Posted by "Woolf, Ross" <Ro...@BMC.com>.
I'm looking for some clarification on the use of field cache in a real time index situation.

We are using Lucene in a real time fashion, but we update our reader via IndexReader.reopen() rather than using the IndexWriter.getReader(); After opening a new reader the old reader is closed.

In the book Lucene in Action (2nd) it mentions something regarding since 2.9.2 you can pass in a subreader instead of the reader if you are reopening a reader so that you only have to load the new segment.  But it is not clear on how all of this works.

My big question is does doing it this way mean that I can reuse the portion of the cache that was created with prior readers?  Or does the whole cache have to be rebuilt each time I do a reopen?  I guess what I am asking is if in a real time situation, since I will be reopening readers regularly, is my cache constantly having to be rebuilt from scratch, or can I use this subreader approach to sequentially add to the cache?

Also if the approach lets me sequentially add to the cache, how exactly do I go about dealing with the subreaders?  All I have used before is my primary reader and as I perform a reopen, how do I know what subreader to pass into the fieldcache?

Any info appreciated.

Re: IndexReader.getSequentialSubReaders() usage in Lucene 2.9+

Posted by Simon Willnauer <si...@googlemail.com>.
Hey Nikolay,

On Thu, May 27, 2010 at 11:00 AM, Nikolay Zamosenchuk
<ni...@exoplatform.com> wrote:
> Hi, Dear colleagues!
> I have one question concerning IndexReader.getSequentialSubReaders()
> and it's usage.

getSequentialSubReaders() was introduced to support Per-Segment Search
in Lucene 2.9. It is used to access the readers a higher level reader
is composed of.
If you have a reader not being composed of a set of other readers this
method does nothing.
>
> Imagine there is a class extending DirectoryReader or MultiReader.
> Usually directory- or multi-reader consists of sub-readers (i.e.
> segment-readers). Is it safe enough to return always null in
> IndexReader.getSequentialSubReaders() ?

Yes it is safe to do so - actually the java doc says that you should
return null if you reader is not composed from child readers.

>
> We have a special Query and Weight classes that relies on custom
> extended IndexReader. If this method (getSequentialSubReaders) is not
> overridden, then our Weight class receives ReadOnlySegmentReader
> instance.

IndexSearcher tries to gather all subreaders from a Top-Level reader
when it is initialized. If you wanna use your top-level reader to be
the only reader used for searches just return null.
> I found that IndexSearcher.search(...) method changed in lucene
> 2.4.1->2.9.2, so subReaders are passed instead of original reader.
> I've also found, that returning null in
> IndexReader.getSequentialSubReaders() will avoid usage of subReaders.
> Seems returning the null helped and tests pessed.
> But there is a question, is it safe enough to return null in
> IndexReader.getSequentialSubReaders() ? Would it break some internal
> logic?

Afaik It would not break any logic though but if you work around per
segment readers you might not be able to benefit from the improvements
made by per segment search. In short: Per-Segment search uses
subreaders (SegementReaders) to reference for instance caches or
filters. If you use your top level reader for searches directly you
force lucene to invalidate all references where the reader is used if
the instance changes. While if you use segment readers you only need
to invalidate the caches of the segments which have changed.

simon


>
> Thanks much!
> --
> Best regards, Nikolay Zamosenchuk
> eXo (UA)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org