You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by David Smiley <ds...@apache.org> on 2024/07/06 02:58:26 UTC

IndexWriter.getReader speed (NRT)

In the context of looking at NRT search in Solr (the so-called
realtime searcher), I was looking at IndexWriter.getReader to see what
it does, as it seems to be the primary method for opening a new view
quickly.  If Lucene has a better entry point for opening a view ASAP,
let me know.  I see that getReader writes a segment.  A very long time
ago in ~2013, I recall Michael Busch, then at Twitter, was
contributing some amazing features to Lucene to search the index
writing buffers in-place.  Do people remember what became of that?
Maybe I'm wrong -- no contribution.  I don't see anything in
CHANGES.txt from Michael that's pertinent.  I didn't yet look in JIRA.
The presentation I saw (in person) by Michael was at Lucene/Solr
Revolution in Boston, 2013 -- recording:
https://www.youtube.com/watch?v=F2CHE4VyB3c

getReader may also merge segments (configured to do so by default
since 9.3), thus making this not very NRT friendly for a subset of
use-cases that a search app may have.  In my experience these relate
to indexing to do conditional updates based on index state or to
return index statistics per batch.  Wouldn't it be useful for
maxFullFlushMergeWaitMillis to be over-rideable as a parameter in case
the caller wishes to get a view without waiting at all for merges but
leaving open other use-cases to use the default/configured time?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: IndexWriter.getReader speed (NRT)

Posted by Adrien Grand <jp...@gmail.com>.
Michael's work was indeed never merged. This approach of searching the
IndexWriter buffer is tempting on paper, but I worry that it would come
with lots of asterisks and be hard to use in practice.
 - Terms and points are not sorted in the IW buffer, so some queries like
range queries would be unsupported or run linear scans.
 - Postings don't have index statistics, skip data or impacts in the IW
buffer, so we would need to cache more data (at the expense of higher
memory usage) or run linear scans.
 - IndexWriter could not release heap from a DWPT if a search is running on
this DWPT.

Letting the caller decide whether to wait for full-flush merges or not on
reopen would require Lucene to track full-flush merges a bit more carefully
but it sounds doable. This is something we would probably take advantage of
in Elasticsearch as we sometimes reopen indexes for internal reasons, where
we don't care about waiting for full-flush merges (while we still want to
wait for full-flush merges for user-facing searches).

On Tue, Jul 30, 2024 at 9:42 PM David Smiley <ds...@apache.org> wrote:

> On Mon, Jul 29, 2024 at 3:51 PM Michael Froh <ms...@gmail.com> wrote:
> >
> > Hi David,
> >
> > Great meeting you at Buzzwords last month!
>
> Nice seeing you too!
>
> ...
> > Adding an overload to IndexWriter.getReader would be pretty easy, but
> that method is package-private. The hairier part probably involves deciding
> which open/openIfChanged methods to overload in DirectoryReader, since I
> think that's where the parameter would ultimately need to be exposed to
> users.
>
> Yes.
>
> Admittedly this isn't something I'm actively seeking right now.  I was
> motivated by Solr's code that has a "realtime" searcher distinction
> internally, in which we try to open a searcher ASAP..  However I
> discovered that in practice there is no practical distinction since
> the regular searcher is almost always the same, thus opening searchers
> sometimes faster isn't worth it.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

-- 
Adrien

Re: IndexWriter.getReader speed (NRT)

Posted by David Smiley <ds...@apache.org>.
On Mon, Jul 29, 2024 at 3:51 PM Michael Froh <ms...@gmail.com> wrote:
>
> Hi David,
>
> Great meeting you at Buzzwords last month!

Nice seeing you too!

...
> Adding an overload to IndexWriter.getReader would be pretty easy, but that method is package-private. The hairier part probably involves deciding which open/openIfChanged methods to overload in DirectoryReader, since I think that's where the parameter would ultimately need to be exposed to users.

Yes.

Admittedly this isn't something I'm actively seeking right now.  I was
motivated by Solr's code that has a "realtime" searcher distinction
internally, in which we try to open a searcher ASAP..  However I
discovered that in practice there is no practical distinction since
the regular searcher is almost always the same, thus opening searchers
sometimes faster isn't worth it.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: IndexWriter.getReader speed (NRT)

Posted by Michael Froh <ms...@gmail.com>.
Hi David,

Great meeting you at Buzzwords last month!

(Sorry for the late reply -- I was on vacation for weeks.)

You can modify maxFullFlushMergeWaitMillis at the IndexWriterConfig level,
but that sets it for any caller who tries to open an IndexReader from the
IndexWriter.

It sounds like you would like an IndexWriter that "normally" merges on
getReader calls (with the default 500ms wait), but can also return a reader
without merging if explicitly requested (in the getReader call).

Adding an overload to IndexWriter.getReader would be pretty easy, but that
method is package-private. The hairier part probably involves deciding
which open/openIfChanged methods to overload in DirectoryReader, since I
think that's where the parameter would ultimately need to be exposed to
users.

Is that a fair assessment of what you're looking for?

Thanks,
Froh

On Fri, Jul 5, 2024 at 8:00 PM David Smiley <ds...@apache.org> wrote:

> In the context of looking at NRT search in Solr (the so-called
> realtime searcher), I was looking at IndexWriter.getReader to see what
> it does, as it seems to be the primary method for opening a new view
> quickly.  If Lucene has a better entry point for opening a view ASAP,
> let me know.  I see that getReader writes a segment.  A very long time
> ago in ~2013, I recall Michael Busch, then at Twitter, was
> contributing some amazing features to Lucene to search the index
> writing buffers in-place.  Do people remember what became of that?
> Maybe I'm wrong -- no contribution.  I don't see anything in
> CHANGES.txt from Michael that's pertinent.  I didn't yet look in JIRA.
> The presentation I saw (in person) by Michael was at Lucene/Solr
> Revolution in Boston, 2013 -- recording:
> https://www.youtube.com/watch?v=F2CHE4VyB3c
>
> getReader may also merge segments (configured to do so by default
> since 9.3), thus making this not very NRT friendly for a subset of
> use-cases that a search app may have.  In my experience these relate
> to indexing to do conditional updates based on index state or to
> return index statistics per batch.  Wouldn't it be useful for
> maxFullFlushMergeWaitMillis to be over-rideable as a parameter in case
> the caller wishes to get a view without waiting at all for merges but
> leaving open other use-cases to use the default/configured time?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>