You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "jpountz (via GitHub)" <gi...@apache.org> on 2024/09/18 08:47:15 UTC
[PR] Make MaxScoreBulkScorer repartition scorers when the min competitive increases. [lucene]
jpountz opened a new pull request, #13800:
URL: https://github.com/apache/lucene/pull/13800
MaxScoreBulkScorer partitions scorers into a set of essential scorers and a set of non-essential scorers, depending on the maximum scores produced by scorers and on the current minimum competitive score. An increase of the minimum competitive score has the potential to yield a more favorable partitioning, but repartitioning can also be expensive.
In order to repartition when necessary while avoiding to repartition too often, this PR tracks the minimum value of the minimum competitive score that would produce a more favorable partitioning, and repartitions scorers whenever the minimum competitive score exceeds this threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make MaxScoreBulkScorer repartition scorers when the min competitive increases. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #13800:
URL: https://github.com/apache/lucene/pull/13800#issuecomment-2357876594
luceneutil on wikibigall:
```
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value
CountTerm 8661.73 (4.4%) 8574.75 (4.0%) -1.0% ( -9% - 7%) 0.450
Prefix3 88.88 (4.6%) 88.20 (4.9%) -0.8% ( -9% - 9%) 0.608
AndHighLow 1039.00 (2.2%) 1032.02 (2.2%) -0.7% ( -4% - 3%) 0.336
IntNRQ 75.99 (13.7%) 75.51 (12.1%) -0.6% ( -23% - 29%) 0.878
Wildcard 46.88 (3.5%) 46.68 (3.6%) -0.4% ( -7% - 6%) 0.705
AndHighMed 250.57 (2.1%) 249.53 (1.9%) -0.4% ( -4% - 3%) 0.508
AndStopWords 30.54 (3.6%) 30.46 (2.8%) -0.3% ( -6% - 6%) 0.791
HighTerm 498.05 (3.1%) 496.92 (3.1%) -0.2% ( -6% - 6%) 0.819
MedTerm 598.12 (3.5%) 598.02 (3.6%) -0.0% ( -6% - 7%) 0.988
OrHighNotHigh 253.10 (3.9%) 253.18 (2.5%) 0.0% ( -6% - 6%) 0.978
OrHighNotMed 331.31 (3.6%) 331.51 (2.8%) 0.1% ( -6% - 6%) 0.954
HighTermMonthSort 3206.14 (3.3%) 3208.40 (2.3%) 0.1% ( -5% - 5%) 0.938
OrNotHighLow 982.98 (2.6%) 983.86 (2.5%) 0.1% ( -4% - 5%) 0.912
OrHighNotLow 413.20 (3.7%) 413.63 (2.9%) 0.1% ( -6% - 7%) 0.922
AndHighHigh 67.54 (2.7%) 67.62 (2.2%) 0.1% ( -4% - 5%) 0.871
OrNotHighHigh 273.51 (4.2%) 273.88 (3.2%) 0.1% ( -7% - 7%) 0.910
LowTerm 899.90 (3.4%) 902.57 (3.0%) 0.3% ( -5% - 6%) 0.768
And3Terms 167.53 (2.9%) 168.23 (1.7%) 0.4% ( -4% - 5%) 0.577
And2Terms2StopWords 156.00 (2.2%) 156.66 (1.7%) 0.4% ( -3% - 4%) 0.486
OrNotHighMed 313.45 (5.2%) 314.92 (4.0%) 0.5% ( -8% - 10%) 0.750
HighTermDayOfYearSort 786.72 (3.8%) 790.72 (3.1%) 0.5% ( -6% - 7%) 0.644
OrHighMed 199.21 (2.9%) 200.38 (2.7%) 0.6% ( -4% - 6%) 0.507
PKLookup 284.43 (2.7%) 286.21 (2.2%) 0.6% ( -4% - 5%) 0.427
CountAndHighHigh 48.00 (1.5%) 48.30 (1.9%) 0.6% ( -2% - 4%) 0.236
OrHighHigh 60.01 (3.9%) 60.40 (4.2%) 0.7% ( -7% - 9%) 0.611
CountAndHighMed 146.39 (1.9%) 147.41 (2.1%) 0.7% ( -3% - 4%) 0.269
HighTermTitleSort 158.66 (3.3%) 159.84 (2.5%) 0.7% ( -4% - 6%) 0.427
OrHighLow 739.91 (1.7%) 746.60 (1.3%) 0.9% ( -2% - 3%) 0.058
OrStopWords 33.69 (5.9%) 34.01 (4.9%) 1.0% ( -9% - 12%) 0.574
Or2Terms2StopWords 157.91 (3.8%) 159.47 (2.8%) 1.0% ( -5% - 7%) 0.352
OrHighRare 264.04 (4.0%) 266.74 (3.8%) 1.0% ( -6% - 9%) 0.407
TermDTSort 341.86 (7.4%) 346.04 (6.2%) 1.2% ( -11% - 16%) 0.571
Or3Terms 166.14 (3.8%) 168.34 (2.9%) 1.3% ( -5% - 8%) 0.216
HighTermTitleBDVSort 11.47 (7.1%) 11.79 (9.2%) 2.8% ( -12% - 20%) 0.281
CountOrHighMed 111.46 (11.2%) 118.04 (11.1%) 5.9% ( -14% - 31%) 0.094
CountOrHighHigh 55.16 (15.2%) 59.24 (14.8%) 7.4% ( -19% - 44%) 0.119
```
This change only helps disjunctions whose queries produce large scoring windows (via `Scorer#advanceShallow`). So it's expected we're not seeing a speedup here since term queries produce scoring windows that include at most the next 128 docs. It does help when clauses are e.g. instances of `ConstantScoreQuery` though, since `ConstantScoreScorer#advanceShallow` returns `NO_MORE_DOCS`. It is good news that we are not observing a slowdown though, despite the more frequent partitioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make MaxScoreBulkScorer repartition scorers when the min competitive increases. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz merged PR #13800:
URL: https://github.com/apache/lucene/pull/13800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make MaxScoreBulkScorer repartition scorers when the min competitive increases. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #13800:
URL: https://github.com/apache/lucene/pull/13800#issuecomment-2411113632
Here's the fix for conjunctions: https://github.com/apache/lucene/pull/13904.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Make MaxScoreBulkScorer repartition scorers when the min competitive increases. [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #13800:
URL: https://github.com/apache/lucene/pull/13800#issuecomment-2404466253
@zhaih Indeed conjunctions need a similar fix, I'll look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org