[GitHub] [lucene-solr] ErickErickson commented on issue #1169: LUCENE-9004: A minor feature and patch -- support deleting vector values and fix segments merging

[GitBox <gi...@apache.org>]

ErickErickson commented on issue #1169: LUCENE-9004: A minor feature and patch -- support deleting vector values and fix segments merging
URL: https://github.com/apache/lucene-solr/pull/1169#issuecomment-583848170
 
 
   Julie:
   
   Moving the conversation about forceMerge over from the JIRA as per Julie.
   
   I can imagine ways to shorten the merge process, but it'll still take quite a long time. My main concern was that I didn't know if the problem Julie was talking about was functional or not. So it sounds like the issue is "just" performance.
   
   Ways to shorten it: First, I'm assuming you're using TieredMergePolicy, which is the default. The forceMerge(1) option _may_ rewrite any given segment multiple times. There's a limit of 30 segments merged at any given time, see maxMergeAtOnceExplicit. So say you have 300 segments, first you'd have 10 merges of 30 segments in the first pass, then another merge of the resulting segments. Each pass is a complete rewrite of the entire index. Depending on the number of segments, there could be more passes. That limit is mainly there so forceMerge doesn't consume too many resources if, say, indexing or searching are going on, but in your case I'd guess you don't care about that. So you could set it to a very large number and get it done in a single pass.
   
   I think that's about the most savings you'd get, I don't think (but haven't measured) whether merging 150 small segments totaling 300G in a single pass is any slower or faster than merging 10 segments totaling 300G, if you wanted to try that you could set maxMergedSegmentMB. That would simply do more merging in the background during indexing to produce fewer, larger segments. Like I said, though, I don't think this will make any difference.
   
   So my guess is that if you bump maxMergeAtOnceExplicit to a very large number, you'll cut your merge time in half (or a third or quarter, or... depending on the number of passes). It'll still take considerable time, but may be acceptable.
   
   Best,
   Erick
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org