You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "vsop-479 (via GitHub)" <gi...@apache.org> on 2024/01/18 09:33:16 UTC
[I] What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges? [lucene]
vsop-479 opened a new issue, #13022:
URL: https://github.com/apache/lucene/issues/13022
### Description
In TieredMergePolicy.doFindMerges, we pick up segments in segment size's descending order, if `hitTooLarge` we need continue to find smaller segments.
If we pick up segments in ascending order, we could terminate when `hitTooLarge`. We may get a speedup when `SegmentInfos` contains a lot of segments.
In this policy, small segments will have a high priority to be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges? [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz closed issue #13022: What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges?
URL: https://github.com/apache/lucene/issues/13022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges? [lucene]
Posted by "vsop-479 (via GitHub)" <gi...@apache.org>.
vsop-479 commented on issue #13022:
URL: https://github.com/apache/lucene/issues/13022#issuecomment-1900060071
Out of curiosity, i measured the method performance in both pick up order, by add a test case in `TestTieredMergePolicy`(without assert):
seg count | baseline(desc pick) | candidate(asc pick) | speedup
-- | -- | -- | --
20 | 2422458 | 2215500 | 8.5%
50 | 2790250 | 2466834 | 11.5%
120 | 5624833 | 3950459 | 29.7%
200 | 11594958 | 7412708 | 36.0%
When the segment count is larger than 100 (which is unusual), there is a stable speedup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges? [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on issue #13022:
URL: https://github.com/apache/lucene/issues/13022#issuecomment-1914190216
> but the main reason may be doStall takes longer with data growing
This sounds plausible. FWIW I don't think that anyone actually performed testing on a modern HDD, values greater than 1 may perform better.
If you don't mind, I'm closing this issue since it looks like we don't want to change the iteration order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges? [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on issue #13022:
URL: https://github.com/apache/lucene/issues/13022#issuecomment-1903944692
Is `findMerges()` a bottleck for your use-case?
FWIW, this way `TieredMergePolicy` works is a feature, it allows it to greedily pack segments into merges in a way that allows going very close to the max merged segment size, since you can always count on the smaller segments to fill the gap. You can no longer easily do this if you walk segments in increasing size order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] What if we pick up segments in segment size's ascending order in TieredMergePolicy.doFindMerges? [lucene]
Posted by "vsop-479 (via GitHub)" <gi...@apache.org>.
vsop-479 commented on issue #13022:
URL: https://github.com/apache/lucene/issues/13022#issuecomment-1905672274
> Is findMerges() a bottleck for your use-case?
No, though we find indexing performance downgrade when our index get too big, but the main reason may be `doStall` takes longer with data growing(we set maxMergeCount to 1 on HDD with ES's recommendation). I will enable `MergeScheduler.verbose` to verify that.
I am just wondering if we can walk segments in increasing size order with some contract.
> You can no longer easily do this if you walk segments in increasing size order.
You are right, thanks for indicating it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org