You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Michael Sokolov (Jira)" <ji...@apache.org> on 2020/03/02 18:01:00 UTC
[jira] [Comment Edited] (LUCENE-8962) Can we merge small segments
during refresh, for faster searching?
[ https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049504#comment-17049504 ]
Michael Sokolov edited comment on LUCENE-8962 at 3/2/20 6:00 PM:
-----------------------------------------------------------------
Tests seemed to pass consistently on master, so I pushed there, but now I cherry-picked to 8x branch and see consistent failures there:
{{ [junit4] Tests with failures [seed: 5036466712CCD5FE]:}}
{{ [junit4] - org.apache.lucene.index.TestIndexWriter.testThreadInterruptDeadlock}}
{{ [junit4] - org.apache.lucene.index.TestIndexWriterMergePolicy.testMergeOnCommit}}
was (Author: sokolov):
Tests seemed to pass consistently on master, so I pushed there, but now I cherry-picked to 8x branch and see consistent failures there:
```
[junit4] Tests with failures [seed: 5036466712CCD5FE]:
[junit4] - org.apache.lucene.index.TestIndexWriter.testThreadInterruptDeadlock
[junit4] - org.apache.lucene.index.TestIndexWriterMergePolicy.testMergeOnCommit
```
> Can we merge small segments during refresh, for faster searching?
> -----------------------------------------------------------------
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael McCandless
> Priority: Major
> Attachments: LUCENE-8962_demo.png
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory segments to disk and open an {{IndexReader}} to search them, and this is typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}} will accumulate write many small segments during {{refresh}} and this then adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if given a little time ... so, could we somehow improve {{IndexWriter'}}s refresh to optionally kick off merge policy to merge segments below some threshold before opening the near-real-time reader? It'd be a bit tricky because while we are waiting for merges, indexing may continue, and new segments may be flushed, but those new segments shouldn't be included in the point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy, and some hackity logic to have the merge policy target small segments just written by refresh, but it's tricky to then open a near-real-time reader, excluding newly flushed but including newly merged segments since the refresh originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org