[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

["Erick Erickson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427642#comment-16427642 ] 

Erick Erickson commented on LUCENE-7976:
----------------------------------------

This is coming together, here's a preliminary patch. It has nocommits and several rough spots/hard-coded numbers, code commented out etc.

I'm putting it up in case anyone interested in this wants to take a look at the _approach_ and poke holes in it. Please raise any concerns but also please don't spend a lot of time on the details before I wrap up things I _know_ will need addressing.

Current state:

0> refactors a bit of findMerges to gather the stats into a separate class as that method was getting quite hard follow. I haven't made use of that new class in forceMerge or expungeDeletes yet.

1> forceMerge and expungeDeletes respect maxMergedSegmentSizeMB

2> regular merging will do "singleton merges" on overly-large segments when they're more than 20% deleted docs. 20% is completely arbitrary, don't quite know the correct numbers yet. That handles the case of a single-segment optimize not getting merged away for a long time.

3> forceMerge will purge all deleted docs. It tries to assemble max-sized segments. Any segments where the live docs are larger than maxMergedSegmentSizeMB get a singleton merge.

4> fixes the annoying bit where segments reported on the admin UI are improperly proportioned

5> expungeDeletes now tries to assemble max sized segments from all segments with > 10% deleted docs. If a segment has > 10% deleted docs _and_ it's liveDocs > maxMergedSegmentSizeMB it gets a singleton merge.

What's left to do:

1> more rigorous testing. So far I've just been looking at the admin UI segments screen and saying "that looks about right".

2> Normal merging rewrites the largest segment too often until it gets to max segment size. I think it also merges dissimilar-sized segments too often.

3> compare the total number of bytes written for one of my test runs between the old and new versions. I'm sure this does more writing, just not sure how much.

4> allow forceMerge to merge down to one segment without having to change solrconfig.xml.

5> perhaps refactor, findMerges, forceMerge and findForcedDeletesMerges to make use of common code.

6> ????

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments
> -------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-7976
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7976
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Major
>         Attachments: LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on disk) handled quite easily in a single Lucene index. This is particularly true as features like docValues move data into MMapDirectory space. The current TMP algorithm allows on the order of 50% deleted documents as per a dev list conversation with Mike McCandless (and his blog here:  https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many TB) solutions like "you need to distribute your collection over more shards" become very costly. Additionally, the tempting "optimize" button exacerbates the issue since once you form, say, a 100G segment (by optimizing/forceMerging) it is not eligible for merging until 97.5G of the docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like <maxAllowedPctDeletedInBigSegments> (no, that's not serious name, suggestions welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with smaller segments to bring the resulting segment up to 5G. If no smaller segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). It would be rewritten into a single segment removing all deleted docs no matter how big it is to start. The 100G example above would be rewritten to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the default would be the same behavior we see now. As it stands now, though, there's no way to recover from an optimize/forceMerge except to re-index from scratch. We routinely see 200G-300G Lucene indexes at this point "in the wild" with 10s of  shards replicated 3 or more times. And that doesn't even include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org