[I] Estimate memory usage for merges [lucene]

["carlosdelest (via GitHub)" <gi...@apache.org>]

carlosdelest opened a new issue, #14225:
URL: https://github.com/apache/lucene/issues/14225

   ### Description
   
   Merges should expose an estimation of the amount of heap memory that will be used for the actual merge to be done.
   
   This information should be exposed as part of the merge information, so merge schedulers can check the amount of memory needed to perform the merge and thus make decisions on throttling merges that would take too much memory.
   
   Related to #14208


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org

Re: [I] Estimate memory usage for merges [lucene]

["jpountz (via GitHub)" <gi...@apache.org>]

jpountz commented on issue #14225:
URL: https://github.com/apache/lucene/issues/14225#issuecomment-2657354802

   For what it's worth, I'm not a fan of throttling merges based on memory usage. Merge throttling is already complicated the way it is, so I'm not too excited about adding more constraints to it and having to figure out how they may interfere with one another.
   
   I would rather look into reducing heap usage of merging vectors, it's not right that merging allocates heap in the order of the size of the merge. And in the meantime, I would rather like the workaround to consist of bounding the maximum size of segments that have vectors and bounding the maximum number of merges that run concurrently (which is what #13883 is about, if a machine has N processors, it makes little sense to ever have more than N/2 running merge threads regardless of the number of open index writers).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org