You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jpountz (via GitHub)" <gi...@apache.org> on 2025/06/20 12:05:10 UTC
[PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
jpountz opened a new pull request, #14823:
URL: https://github.com/apache/lucene/pull/14823
`TieredMergePolicy` currently allows 10 segments per tier. With Lucene being increasingly deployed with separate indexing and search tiers that get updated via segment-based replication, I believe that it would make sense for Lucene to have more aggressive merging defaults, a price that is only paid once on the indexing tier, but that benefits all search nodes that serve queries for this index.
Note that this is still a somewhat conservative default, applications with low latency requirements and low update rates will likely want to go even further, with 4 segments per tier, or even 2.
`BaseMergePolicyTestCase#testSimulateAppendOnly` reports a write amplification increase from 3.4 to 3.8, while `BaseMergePolicyTestCase#testSimulateUpdates` reports a write amplification increase from 4.5 to 4.9. In exchange, the number of segments between the floor and max segment sizes decreases by about 20%.
This should especially help queries that have a high per-segment overhead: PK lookups, point queries, multi-term queries and vector searches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3004639468
To be fair, this chart suggests a quite dramatic degradation over time, but these big drops are mostly due to the benchmark becoming harder by increasing the number of dimensions of vectors (annotation FK) or changing the model (annotation GT), and there are some speedups hidden among these big drops such as a merging speedup (annotation IE).
That said, I agree with your point, we want Lucene to work well on near-realtime use-cases that have high update rates as well, so we can't afford to make indexing and merging absurdly slow just for the sake of having slightly faster search.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz merged PR #14823:
URL: https://github.com/apache/lucene/pull/14823
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3006267677
The right default value for beam width is a little hard to say. Values reported in public vary wildly. Try asking Gemini what is the right default and it refuses to be pinned down! Unfortunately it depends - on the application setup (does it have a fancy second-stage ranker, or do we rely on the vector scores directly for ranking), on the vector data set (its dimensions, the size of the index). We're finding that "high quality" vectors might require lower values for these exploration parameters, and applications that gather lots of semantic hits and then later re-rank them might be less sensitive to lower recall and can tolerate cheaper indexing settings. You can see that these sources recommend values like 16, 64, 128, and 256:
* https://opensearch.org/blog/a-practical-guide-to-selecting-hnsw-hyperparameters/
* https://www.pinecone.io/learn/series/faiss/hnsw/
Maybe a good rule of thumb would be to pick the place where the slope of these latency/recall curves is 45 degrees? But of course that will depend on the units of the chart.
My view is this is something that, at least as a default, we could detune a bit. Maybe we could retreat to 64? But anyway this issue is about merge policy segment counts I think so we should probably open a new issue if we want to change that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3004937275
> but these big drops are mostly due to the benchmark becoming harder by increasing the number of dimensions of vectors (annotation FK) or changing the model (annotation GT), and there are some speedups hidden among these big drops such as a merging speedup (annotation IE).
I don't think that statement is accurate. The first big drop is `DW`. LUCENE-10109: Increase default beam width from 16 to 100.
This one explicitly makes indexing more than 3x slower (from 409 GB/hr -> 140.7 GB/hr). The commit message states "This new default tries to balance good search performance with indexing speed."
Is the balance a good one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3001353043
Here's the trend of HNSW indexing performance in the nightly bench:
<img width="654" alt="Screenshot 2025-06-24 at 1 42 29 PM" src="https://github.com/user-attachments/assets/e6f51f7c-5c76-48ec-9a84-a39c91aeb108" />
You can see why I think wrong tradeoffs are being made. It is trending in the wrong direction and everyone is focused instead on making 1000QPS into 1200QPS or whatever.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #14823:
URL: https://github.com/apache/lucene/pull/14823#discussion_r2164006633
##########
lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java:
##########
@@ -85,7 +85,7 @@ public class TieredMergePolicy extends MergePolicy {
private long maxMergedSegmentBytes = 5 * 1024 * 1024 * 1024L;
private long floorSegmentBytes = 16 * 1024 * 1024L;
- private double segsPerTier = 10.0;
+ private double segsPerTier = 8.0;
Review Comment:
Thanks for catching this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3006456064
Yeah I'm just commenting on overall indexing-vs-search speed perf. As I mentioned above, it isn't intended to block this issue at all: I just hope we are working to have good defaults that take indexing speed into consideration, as it is very slow for HNSW: I imagine this issue will make that even worse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3001141072
My only concern is that it will have a really serious impact on the HNSW indexing/merging.
I know this indexing performance part isn't as fun/sexy as speeding up the queries, but not everyone has thousands of QPS: the queries could probably be slower. It could probably have 1% shittier recall and 10x faster indexing. I'm concerned wrong tradeoffs are made, and I feel like instead a lot of users suffer from the inefficient indexing. This will make that pain worse.
I don't think it should block the change but we should keep an eye on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-2991280830
This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-2993010598
Thanks @mikemccand ! I'll wait a few days before merging to give others a chance to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-3006067996
I'll need @benwtrent or @msokolov to provide an educated answer to this question, but the issue mentions that the previous value also had worse recall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "easyice (via GitHub)" <gi...@apache.org>.
easyice commented on code in PR #14823:
URL: https://github.com/apache/lucene/pull/14823#discussion_r2163420971
##########
lucene/core/src/java/org/apache/lucene/index/TieredMergePolicy.java:
##########
@@ -85,7 +85,7 @@ public class TieredMergePolicy extends MergePolicy {
private long maxMergedSegmentBytes = 5 * 1024 * 1024 * 1024L;
private long floorSegmentBytes = 16 * 1024 * 1024L;
- private double segsPerTier = 10.0;
+ private double segsPerTier = 8.0;
Review Comment:
It seems the default value in the JavaDoc for the `setSegmentsPerTier` method might be outdated？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Decrease TieredMergePolicy's default number of segments per tier to 8. [lucene]
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #14823:
URL: https://github.com/apache/lucene/pull/14823#issuecomment-2991356685
+1, this is a great idea -- more aggessive merging by default makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org