You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jpountz (via GitHub)" <gi...@apache.org> on 2023/12/05 08:49:01 UTC
[I] Add a MergePolicy wrapper that preserves search concurrency? [lucene]
jpountz opened a new issue, #12877:
URL: https://github.com/apache/lucene/issues/12877
### Description
We have an issue about decoupling search concurrency from index geometry (#9721), but this comes with trade-offs as the per-segment bit of search is hard to parallelize. Maybe we should also introduce a merge policy wrapper that tries to preserve a search concurrency of `N` by preventing the creation of segments of more than `maxDoc/N` docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Add a MergePolicy wrapper that preserves search concurrency? [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on issue #12877:
URL: https://github.com/apache/lucene/issues/12877#issuecomment-1845325507
> +1, I like this idea.
I have a vague recollection of you saying you already implemented something like that, am I making this up? (it's quite possible, I struggle to keep lots of stuff in memory!)
> It might be implemented by having TieredMergePolicy dynamically set the max segment size
One potential issue that comes to mind with this approach is that `TieredMergePolicy` gives up on finding balanced merges when they reach the maximum merged segment size and just packs as many segments as possible as long as the sum of their sizes is under the threshold, so this could potentially worsen write amplification?
> Perhaps when the index is tiny it doesn't do the maxDoc/N, but only once enough maxDoc have been index, start enforcing that ...
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Add a MergePolicy wrapper that preserves search concurrency? [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #12877:
URL: https://github.com/apache/lucene/issues/12877#issuecomment-1856109370
> > am I making this up?
>
> Ha! No, you are not hallucinating @jpountz! We do have something like this for Amazon product search -- it's crucial for our usage to keep long-pole query latencies low by maximizing concurrency -- but it might just be as stupid as "setMaxMergedSegmentMB" to something "just right" for our usage. I'll poke around inside and see if our impl is not too embarrassing to share ;)
OK well our (Amazon product search's) implementation is sorta messy: we subclass `TMP` and override `findMerges` to dynamically change `maxMergedSegmentMB` as a function of the total index size at this moment, and then return `super.findMerges()`. It works but it's not so clean. I would prefer for this issue that we make this a first class feature of TMP? Something like `setMinSegmentCount` or `setBalancedSegmentCount` or so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Add a MergePolicy wrapper that preserves search concurrency? [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz closed issue #12877: Add a MergePolicy wrapper that preserves search concurrency?
URL: https://github.com/apache/lucene/issues/12877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Add a MergePolicy wrapper that preserves search concurrency? [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #12877:
URL: https://github.com/apache/lucene/issues/12877#issuecomment-1845431077
> am I making this up?
Ha! No, you are not hallucinating @jpountz! We do have something like this for Amazon product search -- it's crucial for our usage to keep long-pole query latencies low by maximizing concurrency -- but it might just be as stupid as "setMaxMergedSegmentMB" to something "just right" for our usage. I'll poke around inside and see if our impl is not too embarrassing to share ;)
> > It might be implemented by having TieredMergePolicy dynamically set the max segment size
>
> One potential issue that comes to mind with this approach is that `TieredMergePolicy` gives up on finding balanced merges when they reach the maximum merged segment size and just packs as many segments as possible as long as the sum of their sizes is under the threshold, so this could potentially worsen write amplification?
Hmm you're right -- TMP would spend more time doing this "unbalanced packing", though, presumably it would sort of run out of options because it gobbles up the smallish segments aggressively, maybe? Hard to visualize...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Add a MergePolicy wrapper that preserves search concurrency? [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on issue #12877:
URL: https://github.com/apache/lucene/issues/12877#issuecomment-1845276785
+1, I like this idea. It might be implemented by having `TieredMergePolicy` dynamically set the max segment size (in doc count, not just bytes) as a function of total `maxDoc` in the index?
Perhaps when the index is tiny it doesn't do the `maxDoc/N`, but only once enough maxDoc have been index, start enforcing that ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org