[jira] [Created] (LUCENE-10073) Allow very small merges to merge more than segmentsPerTier segments?

["Adrien Grand (Jira)" <ji...@apache.org>]

Adrien Grand created LUCENE-10073:
-------------------------------------

             Summary: Allow very small merges to merge more than segmentsPerTier segments?
                 Key: LUCENE-10073
                 URL: https://issues.apache.org/jira/browse/LUCENE-10073
             Project: Lucene - Core
          Issue Type: Improvement
            Reporter: Adrien Grand


If you are doing lots of concurrent indexing, NRT search regularly publishes many tiny segments, which in-turn pushes a lot of pressure on merging, which needs to constantly merge these tiny segments so that the total number of segments of the index remains under the budget.

In parallel, TieredMergePolicy's behavior is to merge aggressively segments that are below the floor size. The budget of the number of segments allowed in the index is computed as if all segments were larger than the floor size, and merges that only contain segments below the floor size get a perfect skew which guarantees them to get a better score than any merge that contains one or more segments above the floor size.

I'm considering reducing the merging overhead in the NRT case by raising maxMergeAtOnce and allowing merges to merge more than mergeFactor segments as long as the number of merged segments is below maxMergeAtOnce and the merged segment size is below the floor segment size.

Said otherwise, "normal" merges would be allowed to merge up to mergeFactor segments like today, while small merges (size of the merged segment < floor segment bytes) could go up to maxMergeAtOnce.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org