You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "HUSTERGS (via GitHub)" <gi...@apache.org> on 2025/06/21 09:08:32 UTC

[PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

HUSTERGS opened a new pull request, #14827:
URL: https://github.com/apache/lucene/pull/14827

   ### Description
   This PR propose to specialize function `filterCompetitiveHits` when we have exact 2 scorers, in order to reduce float calculation and potential function calls
   
   Luceneutil result on `wikimediumall`  with `searchConcurrency=0`, `taskCountPerCat=5`, `taskRepeatCount=50` after 20 iterations
   
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                         OrHighRare      116.59      (3.1%)      116.96      (2.7%)    0.3% (  -5% -    6%) 0.734
                          OrHighMed       87.75      (2.4%)       88.83      (2.6%)    1.2% (  -3% -    6%) 0.116
                         AndHighMed       67.91      (2.3%)       69.17      (2.2%)    1.9% (  -2% -    6%) 0.009
                        AndHighHigh       27.96      (1.4%)       28.63      (2.0%)    2.4% (  -1% -    5%) 0.000
                         OrHighHigh       26.16      (1.6%)       26.97      (1.6%)    3.1% (   0% -    6%) 0.000
   ```
   
   
   
   <!--
   If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide.
   https://github.com/apache/lucene/blob/main/CONTRIBUTING.md
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-2993494222

   For what it's worth, the reason of this PR is that I find `filterCompetitiveHits` ocuppied about 13% of flamegraph on OrHighHigh query, 
   ![image](https://github.com/user-attachments/assets/b45154f9-a837-40e9-b8a3-49aae5d31e82)
   
   Also, `filterCompetitiveHits` calls `MathUtil.sumUpperBound` in a loop, seems repeatly calculate `MathUtil.sumRelativeErrorBound(numValues)`, (`numValues` is constant within the loop), I tried to optimize this, but it shows no performance difference, maybe `filterCompetitiveHits` is no longer the bottleneck when `numValues` > 2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on code in PR #14827:
URL: https://github.com/apache/lucene/pull/14827#discussion_r2180356164


##########
lucene/core/src/java/org/apache/lucene/search/ScorerUtil.java:
##########
@@ -116,6 +116,25 @@ public int length() {
     }
   }
 
+  /**
+   * Compute a minimum required score, so that (float) MathUtil.sumUpperBound(minRequiredScore +
+   * maxRemainingScore, numScorers) <= minCompetitiveScore. The computed value may not be the
+   * greatest value that meets this condition, which means that we may fail to filter out some docs.
+   * However, this doesn't hurt correctness, it just means that these docs will be filtered out
+   * later, and the extra work required to compute an optimal value would unlikely result in a
+   * speedup.
+   */
+  static double minRequiredScore(
+      double maxRemainingScore, float minCompetitiveScore, int numScorers) {
+    double minRequiredScore = minCompetitiveScore - maxRemainingScore;
+    double subtraction = Math.ulp((double) minCompetitiveScore);
+    while ((float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers)
+        > minCompetitiveScore) {
+      minRequiredScore -= subtraction;
+    }
+    return minRequiredScore;
+  }

Review Comment:
   Of course! I was doing some benchmark about some subtle changes and final performance, will add UT soon



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3040259716

   Nightly benchmarks confirmed the speedup: https://benchmarks.mikemccandless.com/OrStopWords.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3006088726

   FWIW I could confirm a speedup as well with this change on wiibigall.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3032847633

   The seed did not reproduce for me, but I think I understand the problem. The code assumes that if `a + b > c` then `a - ε + b <= c` (`ε > 0`). However this is not true with floating-point arithmetics, it may be that `a + b` and `a - ε + b` round to the same value. So the code assumed that docs had to have a score of at least `a`, but that would mistakenly filter out docs whose score was `a - ε`.
   
   I pushed a fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024748199

   Sorry for the late reply. 
   I've dug a little bit into this issue. There are two problems. 
   The first one is that **docs NOT collected in baseline ARE collected under this patch**, this is actually caused by the initial `double` to `float` cast, so I changed it from `float` to `double`
   The second one is that **docs collected in baseline are NOT collected under this patch**, it turns out the original code actually let docs with score lower than `minCompetitiveScore` been collected, and this patch savely prunes it away. 
   
   Here is a example which can reproduce the two problems above:
   ```java
   package org.apache.lucene;
   
   import org.apache.lucene.util.MathUtil;
   
   public class Run {
   
     public static void main(String[] args) {
   
       {
         // problem that doc collected in patch but not in baseline
         System.out.println("problem that doc collected in patch but not in baseline");
   
         float minCompetitiveScore = 3.5382755f;
         double score = 2.201078414916992d;
         double maxRemainingScore = 1.337196946144104d;
   
         // false, means this can not be collected in baseline
         System.out.println(
             ((float) MathUtil.sumUpperBound(score + maxRemainingScore, 2)) >= minCompetitiveScore);
   
         {
           float minRequiredScore = (float) (minCompetitiveScore - maxRemainingScore);
           while ((float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, 2) > minCompetitiveScore) {
             minRequiredScore = Math.nextDown(minRequiredScore);
           }
   
           // score=2.201078414916992d, minRequiredScore=2.2010784f
           // true, means this will be collected in patch
           System.out.println(score >= minRequiredScore);
         }
   
         {
           double minRequiredScore = (minCompetitiveScore - maxRemainingScore);
           while ((float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, 2) > minCompetitiveScore) {
             minRequiredScore = Math.nextDown(minRequiredScore);
           }
   
           // false, score=2.201078414916992d, minRequiredScoreD=2.2010785341262817d
           System.out.println(score >= minRequiredScore);
         }
       }
   
       System.out.println();
       {
         // problem that doc collected in baseline but not in patch
         System.out.println("problem that doc collected in baseline but not in patch");
   
         float minCompetitiveScore = 7.638806f;
         double score = 7.638805627822876d;
         double maxRemainingScore = 0.0d;
         int numScorers = 33;
   
         double minRequiredScore = (minCompetitiveScore - maxRemainingScore);
         while ((float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers) > minCompetitiveScore) {
           minRequiredScore = Math.nextDown(minRequiredScore);
         }
   
         // false, means this can not be collected by current patch.
         System.out.println(score >= minRequiredScore);
   
         // true, means this can be collected in baseline
         // MathUtil.sumUpperBound(score + maxRemainingScore, numScorers)=7.638805627822984d;
         // (float) MathUtil.sumUpperBound(score + maxRemainingScore, numScorers)=7.638806f == minCompetitiveScore;
         //
         // The original double (before cast to float) is actually smaller than minCompetitiveScore
         // which means we can actually prune this doc safely.
         System.out.println(
             ((float) MathUtil.sumUpperBound(score + maxRemainingScore, numScorers)) >= minCompetitiveScore);
   
       }
     }
   }
   ```
   
   I did another quick **one** iteration full task luceneutils to verify the hit count, this time it still complains about the hit count, but all the diffs are one direction (patch less than basline), which I think is expected behavior:
   ![image](https://github.com/user-attachments/assets/a986668a-8121-41f0-90b1-6888aa1ec489)
   
   And I run another luceneutil  on `wikimediumall` with `searchConcurrency=0, taskCountPerCat=5, taskRepeatCount=50` after 20 iterations:
   
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                         OrHighRare      120.94      (2.7%)      122.37      (1.9%)    1.2% (  -3% -    5%) 0.109
                          OrHighMed       90.74      (3.6%)       92.40      (2.8%)    1.8% (  -4% -    8%) 0.073
                         AndHighMed       69.64      (2.8%)       71.32      (2.1%)    2.4% (  -2% -    7%) 0.002
                        AndHighHigh       28.49      (2.4%)       29.29      (2.1%)    2.8% (  -1% -    7%) 0.000
                         OrHighHigh       26.83      (2.5%)       27.60      (1.9%)    2.9% (  -1% -    7%) 0.000
   ```
   
   The while loop containing call to `Math.nextDown` (which operate on a `double` rather than `float` now) doesn's seem to add much overhead. I locally added some simple print over the iteration count of the loop, they are almost entirely zero.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on code in PR #14827:
URL: https://github.com/apache/lucene/pull/14827#discussion_r2181197645


##########
lucene/core/src/test/org/apache/lucene/search/TestScorerUtil.java:
##########
@@ -93,4 +94,20 @@ public void testLikelyImpactsEnum() throws IOException {
       }
     }
   }
+
+  public void testMinRequiredScore() {
+    double maxRemainingScore = random().nextDouble();
+    float minCompetitiveScore = random().nextFloat();
+    int numScorers = random().nextInt(1, 10);
+
+    double minRequiredScore =
+        ScorerUtil.minRequiredScore(maxRemainingScore, minCompetitiveScore, numScorers);

Review Comment:
   I tried to assert it in a way that relies on the internal implemenation detail of this function, not sure whether it's accecptable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3031917436

   Thank you, we may want to look into making this better in follow-ups, but this looks good enough for me so I merged. :+1:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3033680943

   I worked on improving the `ScorerUtil` test so that it would catch this problem. It helped me find another problem. I pushed directly to the branch. I think we're good now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #14827:
URL: https://github.com/apache/lucene/pull/14827#discussion_r2178335968


##########
lucene/CHANGES.txt:
##########
@@ -194,6 +194,8 @@ Optimizations
 
 * GITHUB#14854: Using BatchScoreBulkScorer on CombinedFieldQuery. (Ge Song)
 
+* GITHUB#14827: Specialize filterCompetitiveHits when have exact 2 clauses. (Ge Sone)

Review Comment:
   Let's update changes since it now applies to any number of clauses?



##########
lucene/core/src/java/org/apache/lucene/search/ScorerUtil.java:
##########
@@ -128,15 +128,22 @@ static void filterCompetitiveHits(
       double maxRemainingScore,
       float minCompetitiveScore,
       int numScorers) {
-    if ((float) MathUtil.sumUpperBound(maxRemainingScore, numScorers) >= minCompetitiveScore) {
+    // Compute minRequiredScore as the greatest float value so that (float)
+    // MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers) <=
+    // minCompetitiveScore.
+    double minRequiredScore = minCompetitiveScore - maxRemainingScore;
+    while ((float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers)
+        > minCompetitiveScore) {
+      minRequiredScore = Math.nextDown(minRequiredScore);

Review Comment:
   I suspect that this would always converge quickly in practice, but the paranoid me would prefer subtracting an ulp of `minCompetitiveScore` to make sure it always converges quickly.
   ```suggestion
         minRequiredScore -= Math.ulp((double) minCompetitiveScore);
   ```
   
   (Many adjacent values may round to the same value when summed up with `maxRemainingScore` because of how floating point sums round, subtracting an ulp of minCompetitiveScore forces going to a value whose sum with `maxRemainingScore` is lower.)
   



##########
lucene/core/src/java/org/apache/lucene/search/ScorerUtil.java:
##########
@@ -128,15 +128,22 @@ static void filterCompetitiveHits(
       double maxRemainingScore,
       float minCompetitiveScore,
       int numScorers) {
-    if ((float) MathUtil.sumUpperBound(maxRemainingScore, numScorers) >= minCompetitiveScore) {
+    // Compute minRequiredScore as the greatest float value so that (float)
+    // MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers) <=
+    // minCompetitiveScore.

Review Comment:
   The comment needs updating (especially as the previous version that I suggested did not actually compute the greatest float that meets this condition), something like:
   
   ```
   Compute a minimum required score, so that
   (float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers) <= minCompetitiveScore. 
   The computed value may not be the greatest value that meets this condition, which means that we may fail to filter
   out some docs. However, this doesn't hurt correctness, it just means that these docs will be filtered out later, and
   the extra work required to compute an optimal value would unlikely result in a speedup.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024763324

   > I played a bit more with your change, and it looks like we could make things even further as a follow-up through vectorization by using `(Int|Float)Vector#compress`.
   
   Thanks for your advice!  I think i understand what you mean, will try to find a proper way to utilize the vectorization these days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3028333073

   FWIW, I was a little bit worried about the cost of `Math.ulp` (compared to `Math.nextDown`), and also not sure wheather we should calculate the `Math.ulp` first before the while loop, so I did some comparation based on luceneutil. Here is the result if it helps
   
   <details>
   <summary> final patch vs baseline </summary>
   
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                          CountTerm     6770.90      (6.8%)     6583.92      (7.3%)   -2.8% ( -15% -   12%) 0.218
         FilteredOr2Terms2StopWords       65.75      (6.9%)       64.68      (6.3%)   -1.6% ( -13% -   12%) 0.438
                         TermDTSort      188.30      (4.4%)      186.09      (4.5%)   -1.2% (  -9% -    8%) 0.401
                            Term10K      577.54      (7.5%)      570.96      (7.5%)   -1.1% ( -14% -   14%) 0.629
                               Term      577.73      (7.5%)      571.15      (7.4%)   -1.1% ( -14% -   14%) 0.628
                      TermMonthSort     2442.28      (3.3%)     2414.48      (4.7%)   -1.1% (  -8% -    7%) 0.372
                 CombinedAndHighMed       28.53      (5.7%)       28.21      (5.5%)   -1.1% ( -11% -   10%) 0.524
                          TermB1M1P      578.27      (7.4%)      571.77      (7.5%)   -1.1% ( -14% -   14%) 0.633
                  CombinedOrHighMed       27.94      (7.0%)       27.64      (6.6%)   -1.1% ( -13% -   13%) 0.619
                  FilteredOrHighMed       53.15      (5.1%)       52.59      (4.9%)   -1.1% ( -10% -    9%) 0.500
                             Term1M      577.12      (7.5%)      571.38      (7.4%)   -1.0% ( -14% -   15%) 0.672
                   FilteredOr3Terms       58.49      (4.8%)       57.96      (4.5%)   -0.9% (  -9% -    8%) 0.541
                            TermB1M      577.53      (7.3%)      572.52      (7.6%)   -0.9% ( -14% -   15%) 0.712
                             IntSet      391.58      (4.5%)      388.40      (4.8%)   -0.8% (  -9% -    8%) 0.582
                  FilteredAnd3Terms      129.69      (3.4%)      128.71      (2.6%)   -0.8% (  -6% -    5%) 0.433
                            Term100      577.00      (7.4%)      572.67      (7.3%)   -0.8% ( -14% -   15%) 0.748
                      TermTitleSort       69.16      (3.7%)       68.80      (4.9%)   -0.5% (  -8% -    8%) 0.707
                       CombinedTerm       14.62      (4.9%)       14.55      (5.6%)   -0.5% ( -10% -   10%) 0.770
                     CountOrHighMed       95.39      (2.2%)       94.93      (2.2%)   -0.5% (  -4% -    3%) 0.488
                           Wildcard       58.16      (3.6%)       57.89      (3.5%)   -0.5% (  -7% -    6%) 0.669
                           SpanNear        3.11      (4.0%)        3.10      (4.3%)   -0.4% (  -8% -    8%) 0.750
                    FilteredPrefix3       91.64      (3.2%)       91.27      (3.2%)   -0.4% (  -6% -    6%) 0.691
                       FilteredTerm       85.80      (5.3%)       85.48      (4.7%)   -0.4% (  -9% -   10%) 0.813
                   IntervalsOrdered        2.98      (3.8%)        2.97      (4.1%)   -0.4% (  -7% -    7%) 0.770
                    CountAndHighMed       92.52      (2.8%)       92.19      (2.6%)   -0.3% (  -5% -    5%) 0.683
                             IntNRQ       48.55      (1.9%)       48.39      (1.7%)   -0.3% (  -3% -    3%) 0.571
                         DismaxTerm      625.93      (6.1%)      624.14      (5.9%)   -0.3% ( -11% -   12%) 0.880
                 FilteredOrHighHigh       17.88      (3.2%)       17.83      (3.3%)   -0.3% (  -6% -    6%) 0.784
                       SloppyPhrase        1.45      (4.6%)        1.45      (4.0%)   -0.3% (  -8% -    8%) 0.839
                            Prefix3       98.03      (3.5%)       97.78      (3.5%)   -0.3% (  -7% -    6%) 0.816
                    CountOrHighHigh       63.25      (2.1%)       63.11      (2.5%)   -0.2% (  -4% -    4%) 0.761
                     FilteredIntNRQ       48.33      (2.1%)       48.25      (1.7%)   -0.2% (  -3% -    3%) 0.777
                CountFilteredIntNRQ       22.23      (1.9%)       22.20      (1.7%)   -0.2% (  -3% -    3%) 0.773
                CountFilteredOrMany        6.02      (1.5%)        6.01      (1.8%)   -0.2% (  -3% -    3%) 0.758
                   CountAndHighHigh       61.53      (1.3%)       61.45      (1.7%)   -0.1% (  -3% -    2%) 0.783
                   AndMedOrHighHigh       20.70      (4.3%)       20.70      (4.1%)    0.0% (  -8% -    8%) 0.993
                CountFilteredPhrase       11.57      (2.1%)       11.57      (2.1%)    0.0% (  -4% -    4%) 0.945
                             Phrase        9.82      (2.6%)        9.83      (2.3%)    0.1% (  -4% -    5%) 0.918
            CountFilteredOrHighHigh       25.13      (1.2%)       25.16      (1.5%)    0.1% (  -2% -    2%) 0.829
                     FilteredOrMany        5.01      (2.9%)        5.01      (2.6%)    0.1% (  -5% -    5%) 0.916
             CountFilteredOrHighMed       29.53      (1.3%)       29.57      (1.4%)    0.1% (  -2% -    2%) 0.766
                    AndHighOrMedMed       18.23      (3.9%)       18.26      (2.6%)    0.2% (  -6% -    6%) 0.882
                    DismaxOrHighMed       64.34      (8.7%)       64.46      (8.3%)    0.2% ( -15% -   18%) 0.943
                         OrHighRare      121.70      (4.4%)      121.95      (4.5%)    0.2% (  -8% -    9%) 0.882
                             Fuzzy1       49.44      (4.3%)       49.60      (4.0%)    0.3% (  -7% -    9%) 0.809
                        CountOrMany        6.36      (1.8%)        6.38      (2.5%)    0.3% (  -3% -    4%) 0.640
                  TermDayOfYearSort      343.74      (1.3%)      344.99      (1.6%)    0.4% (  -2% -    3%) 0.433
                     FilteredPhrase       12.55      (3.0%)       12.60      (2.1%)    0.4% (  -4% -    5%) 0.631
                             Fuzzy2       44.71      (2.8%)       44.88      (2.5%)    0.4% (  -4% -    5%) 0.637
                        CountPhrase        3.28      (4.6%)        3.30      (4.7%)    0.4% (  -8% -   10%) 0.770
                          OrHighMed       86.40     (13.5%)       86.84     (13.4%)    0.5% ( -23% -   31%) 0.904
                CombinedAndHighHigh        7.17      (2.3%)        7.22      (2.2%)    0.6% (  -3% -    5%) 0.395
                 CombinedOrHighHigh        7.01      (4.7%)        7.05      (5.1%)    0.6% (  -8% -   10%) 0.683
               FilteredAndStopWords       11.80      (3.8%)       11.89      (3.1%)    0.7% (  -5% -    7%) 0.510
                            Respell       43.59      (3.2%)       43.92      (1.4%)    0.8% (  -3% -    5%) 0.333
                FilteredOrStopWords       10.92      (3.0%)       11.01      (2.2%)    0.8% (  -4% -    6%) 0.362
                         AndHighMed       66.88      (9.2%)       67.46      (9.1%)    0.9% ( -16% -   21%) 0.765
                FilteredAndHighHigh       14.95      (3.5%)       15.10      (3.2%)    1.0% (  -5% -    7%) 0.353
                   DismaxOrHighHigh       45.06      (7.9%)       45.58      (7.6%)    1.2% ( -13% -   18%) 0.639
                And2Terms2StopWords       71.10     (10.3%)       72.31     (10.3%)    1.7% ( -17% -   24%) 0.604
        FilteredAnd2Terms2StopWords       72.77      (7.7%)       74.03      (7.5%)    1.7% ( -12% -   18%) 0.474
                        AndHighHigh       26.25     (10.1%)       26.71     (10.0%)    1.7% ( -16% -   24%) 0.581
                 FilteredAndHighMed       43.31      (4.5%)       44.07      (4.1%)    1.8% (  -6% -   10%) 0.199
                         OrHighHigh       24.55     (12.9%)       25.19     (13.2%)    2.6% ( -20% -   32%) 0.527
                 Or2Terms2StopWords       71.62     (12.1%)       73.89     (11.8%)    3.2% ( -18% -   30%) 0.403
                          And3Terms       83.39      (8.5%)       87.53      (7.7%)    5.0% ( -10% -   23%) 0.053
                             OrMany        5.42      (7.8%)        5.73      (5.0%)    5.6% (  -6% -   19%) 0.006
                           Or3Terms       73.96     (11.5%)       78.67     (11.4%)    6.4% ( -14% -   33%) 0.078
                        OrStopWords        9.31     (11.6%)       10.08     (12.2%)    8.3% ( -13% -   36%) 0.027
                       AndStopWords        8.85      (9.2%)        9.67      (8.4%)    9.2% (  -7% -   29%) 0.001
   WARNING: cat=Fuzzy1: hit counts differ: 11543+ vs 11524+
   WARNING: cat=Fuzzy2: hit counts differ: 12724+ vs 12716+
   WARNING: cat=Or2Terms2StopWords: hit counts differ: 14986+ vs 14985+
   ```
   
   </details>
   
   <details>
   <summary> cached <code>Math.ulp</code> (baseline) vs  <code>Math.nextDown</code> (candidate) </summary>
   
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                         OrHighRare      122.65      (4.9%)      120.37      (4.8%)   -1.9% ( -11% -    8%) 0.229
                       CombinedTerm       14.67      (4.5%)       14.55      (3.9%)   -0.8% (  -8% -    7%) 0.527
                     FilteredPhrase       12.73      (1.8%)       12.63      (2.0%)   -0.8% (  -4% -    3%) 0.205
                    FilteredPrefix3       92.09      (2.2%)       91.41      (2.8%)   -0.7% (  -5% -    4%) 0.356
                CombinedAndHighHigh        7.26      (1.8%)        7.22      (2.8%)   -0.6% (  -5% -    4%) 0.449
                CountFilteredIntNRQ       22.37      (2.4%)       22.25      (2.2%)   -0.5% (  -4% -    4%) 0.479
                    AndHighOrMedMed       18.60      (2.7%)       18.52      (3.3%)   -0.4% (  -6% -    5%) 0.644
             CountFilteredOrHighMed       29.92      (2.0%)       29.80      (1.7%)   -0.4% (  -3% -    3%) 0.488
                   DismaxOrHighHigh       46.43      (5.9%)       46.27      (5.7%)   -0.4% ( -11% -   11%) 0.843
            CountFilteredOrHighHigh       25.45      (1.8%)       25.37      (1.6%)   -0.3% (  -3% -    3%) 0.549
                     FilteredIntNRQ       48.27      (2.1%)       48.12      (2.4%)   -0.3% (  -4% -    4%) 0.661
                            Respell       43.46      (2.1%)       43.33      (2.0%)   -0.3% (  -4% -    3%) 0.646
                       SloppyPhrase        1.44      (4.6%)        1.44      (5.0%)   -0.3% (  -9% -    9%) 0.847
                 CombinedOrHighHigh        7.08      (4.5%)        7.06      (4.6%)   -0.3% (  -9% -    9%) 0.842
                        AndHighHigh       26.93      (9.2%)       26.85      (9.0%)   -0.3% ( -16% -   19%) 0.922
                        CountOrMany        6.41      (2.9%)        6.40      (2.6%)   -0.3% (  -5% -    5%) 0.768
                            Prefix3       98.30      (2.8%)       98.07      (3.7%)   -0.2% (  -6% -    6%) 0.827
                             IntSet      398.07      (5.0%)      397.31      (4.7%)   -0.2% (  -9% -    9%) 0.901
                       FilteredTerm       89.51      (4.3%)       89.37      (4.4%)   -0.2% (  -8% -    9%) 0.908
                      TermTitleSort       69.24      (3.9%)       69.16      (4.0%)   -0.1% (  -7% -    8%) 0.928
                  TermDayOfYearSort      343.02      (1.6%)      342.64      (1.0%)   -0.1% (  -2% -    2%) 0.796
                          And3Terms       89.96      (6.1%)       89.89      (6.1%)   -0.1% ( -11% -   12%) 0.964
                           Wildcard       58.53      (2.6%)       58.48      (3.5%)   -0.1% (  -5% -    6%) 0.936
                             IntNRQ       48.59      (2.2%)       48.56      (2.0%)   -0.1% (  -4% -    4%) 0.918
                   CountAndHighHigh       61.94      (1.5%)       61.90      (1.6%)   -0.1% (  -3% -    3%) 0.911
                         OrHighHigh       25.88      (9.1%)       25.87      (9.0%)   -0.0% ( -16% -   19%) 0.996
                FilteredAndHighHigh       15.19      (2.3%)       15.19      (2.5%)    0.0% (  -4% -    4%) 0.995
                         DismaxTerm      642.57      (5.7%)      642.60      (5.7%)    0.0% ( -10% -   12%) 0.997
                FilteredOrStopWords       11.18      (1.8%)       11.19      (2.6%)    0.0% (  -4% -    4%) 0.978
                   AndMedOrHighHigh       21.04      (3.5%)       21.04      (3.5%)    0.0% (  -6% -    7%) 0.985
               FilteredAndStopWords       11.98      (2.5%)       11.98      (2.5%)    0.0% (  -4% -    5%) 0.970
                          OrHighMed       91.32      (8.8%)       91.36      (8.5%)    0.0% ( -15% -   19%) 0.987
                CountFilteredOrMany        6.07      (1.8%)        6.07      (1.9%)    0.1% (  -3% -    3%) 0.922
                       AndStopWords        9.63      (6.2%)        9.63      (6.3%)    0.1% ( -11% -   13%) 0.976
                             Term1M      587.59      (7.6%)      588.02      (6.8%)    0.1% ( -13% -   15%) 0.975
                 CombinedAndHighMed       29.45      (4.2%)       29.47      (4.5%)    0.1% (  -8% -    9%) 0.956
                   IntervalsOrdered        2.93      (3.3%)        2.94      (2.4%)    0.1% (  -5% -    5%) 0.915
                CountFilteredPhrase       11.49      (2.1%)       11.50      (1.8%)    0.1% (  -3% -    4%) 0.865
                          CountTerm     6878.31      (5.8%)     6886.78      (5.5%)    0.1% ( -10% -   12%) 0.945
                        OrStopWords       10.27      (7.9%)       10.28      (8.3%)    0.1% ( -14% -   17%) 0.960
                    DismaxOrHighMed       66.86      (5.8%)       66.96      (5.6%)    0.1% ( -10% -   12%) 0.937
                         AndHighMed       68.99      (8.0%)       69.09      (8.1%)    0.1% ( -14% -   17%) 0.955
                    CountOrHighHigh       63.98      (2.0%)       64.08      (2.1%)    0.2% (  -3% -    4%) 0.806
                           Or3Terms       81.49      (6.9%)       81.63      (6.8%)    0.2% ( -12% -   14%) 0.938
                               Term      586.86      (7.7%)      587.87      (7.0%)    0.2% ( -13% -   16%) 0.941
                             Phrase        9.85      (2.3%)        9.87      (3.4%)    0.2% (  -5% -    5%) 0.843
                            Term100      587.22      (7.6%)      588.35      (7.2%)    0.2% ( -13% -   16%) 0.934
                          TermB1M1P      586.68      (7.7%)      588.21      (7.0%)    0.3% ( -13% -   16%) 0.910
                  CombinedOrHighMed       28.78      (5.6%)       28.87      (5.1%)    0.3% (  -9% -   11%) 0.862
                 FilteredAndHighMed       44.81      (3.2%)       44.97      (3.1%)    0.4% (  -5% -    6%) 0.715
                         TermDTSort      193.95      (4.4%)      194.66      (4.2%)    0.4% (  -7% -    9%) 0.787
                 FilteredOrHighHigh       18.21      (2.6%)       18.29      (2.9%)    0.5% (  -4% -    6%) 0.593
                            TermB1M      586.34      (7.6%)      589.19      (7.0%)    0.5% ( -13% -   16%) 0.834
                             Fuzzy2       45.79      (2.2%)       46.01      (2.6%)    0.5% (  -4% -    5%) 0.510
                           SpanNear        3.04      (5.3%)        3.06      (5.1%)    0.5% (  -9% -   11%) 0.757
                    CountAndHighMed       92.07      (3.7%)       92.61      (3.4%)    0.6% (  -6% -    8%) 0.605
                             Fuzzy1       51.48      (3.5%)       51.78      (3.8%)    0.6% (  -6% -    8%) 0.610
                 Or2Terms2StopWords       79.27      (8.1%)       79.76      (7.7%)    0.6% ( -14% -   17%) 0.808
        FilteredAnd2Terms2StopWords       77.97      (6.1%)       78.46      (5.8%)    0.6% ( -10% -   13%) 0.739
                  FilteredAnd3Terms      131.77      (2.9%)      132.61      (2.6%)    0.6% (  -4% -    6%) 0.464
                And2Terms2StopWords       77.57      (8.4%)       78.06      (8.0%)    0.6% ( -14% -   18%) 0.806
                  FilteredOrHighMed       54.67      (4.1%)       55.02      (4.2%)    0.6% (  -7% -    9%) 0.622
                            Term10K      585.60      (7.6%)      589.42      (7.0%)    0.7% ( -12% -   16%) 0.777
                     FilteredOrMany        5.11      (2.6%)        5.14      (2.3%)    0.7% (  -4% -    5%) 0.370
                      TermMonthSort     2472.84      (3.7%)     2490.38      (4.5%)    0.7% (  -7% -    9%) 0.584
                     CountOrHighMed       95.51      (3.2%)       96.23      (2.9%)    0.8% (  -5% -    6%) 0.425
                             OrMany        5.77      (5.3%)        5.81      (4.9%)    0.8% (  -8% -   11%) 0.616
                        CountPhrase        3.24      (5.2%)        3.26      (4.2%)    0.8% (  -8% -   10%) 0.581
                   FilteredOr3Terms       60.14      (4.0%)       60.72      (3.8%)    1.0% (  -6% -    9%) 0.431
         FilteredOr2Terms2StopWords       68.59      (5.7%)       69.35      (5.7%)    1.1% (  -9% -   13%) 0.537
   ```
   
   </details>
   
   <details>
   <summary> cached <code>Math.ulp</code> vs non-cached  </summary>
   
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                        AndHighHigh       27.11      (3.5%)       26.72      (9.8%)   -1.4% ( -14% -   12%) 0.538
                       SloppyPhrase        1.44      (4.0%)        1.43      (3.8%)   -1.3% (  -8% -    6%) 0.310
                CountFilteredPhrase       11.66      (2.0%)       11.54      (1.7%)   -1.0% (  -4% -    2%) 0.079
                        CountPhrase        3.34      (1.7%)        3.31      (3.1%)   -1.0% (  -5% -    3%) 0.226
                       AndStopWords        9.69      (4.3%)        9.61      (7.1%)   -0.9% ( -11% -   11%) 0.647
                   AndMedOrHighHigh       20.97      (2.5%)       20.83      (4.6%)   -0.7% (  -7% -    6%) 0.541
                         AndHighMed       68.38      (3.3%)       68.08      (9.3%)   -0.4% ( -12% -   12%) 0.840
                             Phrase        9.72      (2.4%)        9.68      (2.1%)   -0.4% (  -4% -    4%) 0.588
                  TermDayOfYearSort      343.57      (1.6%)      342.85      (1.8%)   -0.2% (  -3% -    3%) 0.704
                     FilteredIntNRQ       48.19      (1.2%)       48.12      (1.8%)   -0.1% (  -3% -    2%) 0.756
                             IntNRQ       48.40      (1.3%)       48.35      (1.8%)   -0.1% (  -3% -    2%) 0.816
                            Respell       44.00      (2.4%)       43.97      (2.1%)   -0.1% (  -4% -    4%) 0.907
                CountFilteredOrMany        6.00      (2.1%)        5.99      (1.8%)   -0.1% (  -3% -    3%) 0.923
                           SpanNear        3.14      (3.6%)        3.14      (4.0%)    0.0% (  -7% -    7%) 0.998
                       CombinedTerm       14.70      (3.6%)       14.70      (3.1%)    0.0% (  -6% -    6%) 0.995
                    CountOrHighHigh       62.87      (2.5%)       62.94      (2.4%)    0.1% (  -4% -    5%) 0.889
                CombinedAndHighHigh        7.27      (1.7%)        7.28      (2.2%)    0.1% (  -3% -    4%) 0.858
                          And3Terms       88.51      (3.6%)       88.62      (7.9%)    0.1% ( -11% -   12%) 0.950
                   IntervalsOrdered        2.96      (3.1%)        2.97      (3.2%)    0.1% (  -5% -    6%) 0.888
                        OrStopWords       10.20      (9.6%)       10.22      (9.3%)    0.2% ( -17% -   21%) 0.956
                             OrMany        5.82      (3.4%)        5.83      (5.0%)    0.2% (  -7% -    8%) 0.899
                   CountAndHighHigh       61.27      (1.7%)       61.37      (1.6%)    0.2% (  -3% -    3%) 0.742
                        CountOrMany        6.33      (3.3%)        6.34      (2.3%)    0.2% (  -5% -    6%) 0.838
                     CountOrHighMed       95.26      (2.5%)       95.50      (2.2%)    0.2% (  -4% -    5%) 0.737
                CountFilteredIntNRQ       22.12      (1.3%)       22.21      (1.6%)    0.4% (  -2% -    3%) 0.410
                     FilteredPhrase       12.44      (2.4%)       12.49      (2.0%)    0.4% (  -3% -    4%) 0.561
                    CountAndHighMed       92.69      (2.8%)       93.06      (2.4%)    0.4% (  -4% -    5%) 0.622
            CountFilteredOrHighHigh       25.01      (1.8%)       25.13      (1.7%)    0.5% (  -3% -    4%) 0.409
                 CombinedOrHighHigh        7.18      (4.0%)        7.21      (3.4%)    0.5% (  -6% -    8%) 0.643
                    AndHighOrMedMed       18.40      (2.5%)       18.50      (3.1%)    0.5% (  -4% -    6%) 0.543
             CountFilteredOrHighMed       29.41      (1.8%)       29.57      (2.2%)    0.6% (  -3% -    4%) 0.382
                  FilteredAnd3Terms      129.79      (2.6%)      130.56      (3.2%)    0.6% (  -5% -    6%) 0.526
                         OrHighHigh       25.48      (9.7%)       25.65      (9.7%)    0.6% ( -17% -   22%) 0.837
                            Term10K      571.75      (4.7%)      575.74      (8.0%)    0.7% ( -11% -   14%) 0.735
                FilteredOrStopWords       11.01      (2.2%)       11.09      (2.5%)    0.8% (  -3% -    5%) 0.283
                   DismaxOrHighHigh       45.46      (6.0%)       45.83      (6.4%)    0.8% ( -10% -   14%) 0.681
                 FilteredAndHighMed       44.08      (3.0%)       44.45      (4.4%)    0.8% (  -6% -    8%) 0.492
                      TermMonthSort     2383.55      (3.6%)     2403.84      (5.0%)    0.9% (  -7% -    9%) 0.535
                             Term1M      571.35      (4.8%)      576.33      (7.9%)    0.9% ( -11% -   14%) 0.673
                               Term      572.59      (4.8%)      577.59      (8.1%)    0.9% ( -11% -   14%) 0.679
                            TermB1M      571.63      (4.6%)      577.08      (8.1%)    1.0% ( -11% -   14%) 0.646
                            Term100      571.33      (4.8%)      576.88      (8.1%)    1.0% ( -11% -   14%) 0.646
                             IntSet      386.36      (3.7%)      390.32      (4.2%)    1.0% (  -6% -    9%) 0.413
               FilteredAndStopWords       11.79      (3.4%)       11.91      (3.3%)    1.1% (  -5% -    8%) 0.323
                         OrHighRare      117.89      (4.1%)      119.16      (4.5%)    1.1% (  -7% -   10%) 0.433
                    DismaxOrHighMed       64.77      (6.9%)       65.47      (7.1%)    1.1% ( -12% -   16%) 0.625
                FilteredAndHighHigh       14.93      (3.1%)       15.10      (3.2%)    1.1% (  -5% -    7%) 0.264
                          TermB1M1P      570.76      (4.9%)      577.30      (8.0%)    1.1% ( -11% -   14%) 0.586
                     FilteredOrMany        5.04      (2.3%)        5.10      (2.4%)    1.2% (  -3% -    6%) 0.106
                         DismaxTerm      620.66      (4.3%)      628.51      (5.4%)    1.3% (  -8% -   11%) 0.412
                 FilteredOrHighHigh       17.78      (3.2%)       18.00      (3.1%)    1.3% (  -4% -    7%) 0.201
                      TermTitleSort       69.48      (5.1%)       70.40      (4.4%)    1.3% (  -7% -   11%) 0.380
                           Or3Terms       79.24      (8.1%)       80.29      (8.3%)    1.3% ( -13% -   19%) 0.608
                             Fuzzy2       45.38      (2.2%)       46.00      (2.8%)    1.4% (  -3% -    6%) 0.081
                          OrHighMed       87.94     (10.1%)       89.31     (10.4%)    1.6% ( -17% -   24%) 0.629
                         TermDTSort      186.32      (4.1%)      189.37      (3.9%)    1.6% (  -6% -   10%) 0.199
                          CountTerm     6527.34      (4.5%)     6639.20      (6.2%)    1.7% (  -8% -   12%) 0.315
                            Prefix3       98.71      (3.3%)      100.50      (3.8%)    1.8% (  -5% -    9%) 0.106
                 CombinedAndHighMed       28.34      (5.2%)       28.91      (5.5%)    2.0% (  -8% -   13%) 0.232
        FilteredAnd2Terms2StopWords       74.35      (6.4%)       75.85      (7.7%)    2.0% ( -11% -   17%) 0.365
                           Wildcard       57.69      (3.8%)       58.85      (4.4%)    2.0% (  -5% -   10%) 0.117
                   FilteredOr3Terms       57.92      (4.6%)       59.17      (4.7%)    2.2% (  -6% -   12%) 0.143
                             Fuzzy1       50.18      (3.4%)       51.31      (4.5%)    2.3% (  -5% -   10%) 0.074
                  FilteredOrHighMed       52.48      (5.1%)       53.68      (4.8%)    2.3% (  -7% -   12%) 0.147
                    FilteredPrefix3       91.96      (3.0%)       94.13      (3.8%)    2.4% (  -4% -    9%) 0.030
                       FilteredTerm       85.14      (5.1%)       87.42      (5.7%)    2.7% (  -7% -   14%) 0.118
                  CombinedOrHighMed       27.90      (6.1%)       28.65      (5.7%)    2.7% (  -8% -   15%) 0.150
                 Or2Terms2StopWords       74.51      (9.6%)       76.63     (10.3%)    2.8% ( -15% -   25%) 0.366
                And2Terms2StopWords       72.82      (8.2%)       74.91     (10.8%)    2.9% ( -14% -   23%) 0.343
         FilteredOr2Terms2StopWords       64.55      (6.7%)       66.61      (6.9%)    3.2% (  -9% -   17%) 0.139
   ```
   
   </details>
   
   The result shows there is no significant difference between these subtle changes, so I think it's fine to just leave it as is. BTW, I repeat the new UT 100000 times, all passed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024674923

   This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz merged PR #14827:
URL: https://github.com/apache/lucene/pull/14827


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3019121418

   Thinking a bit more about this problem, finding the smallest float that meets the condition may not always be cheap. However, finding a good-enough float should be easy, and it's fine if the hit count changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "dweiss (via GitHub)" <gi...@apache.org>.
dweiss commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3032637198

   Builds are failing after this has been merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024754241

   > The while loop containing call to Math.nextDown (which operate on a double rather than float now) doesn's seem to add much overhead
   
   I'll run a full task luceneutil to verify the performance impact tonight


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024606817

   This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024988818

   luceneutil result under same setup with lasted code:
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                            TermB1M      571.58      (4.4%)      560.31      (5.7%)   -2.0% ( -11% -    8%) 0.219
                          TermB1M1P      571.38      (4.4%)      560.76      (5.6%)   -1.9% ( -11% -    8%) 0.243
                             Term1M      570.90      (4.2%)      560.56      (5.8%)   -1.8% ( -11% -    8%) 0.260
                            Term10K      570.64      (4.4%)      560.95      (5.9%)   -1.7% ( -11% -    8%) 0.301
                       FilteredTerm       84.77      (4.2%)       83.39      (2.7%)   -1.6% (  -8% -    5%) 0.146
                               Term      571.14      (4.3%)      561.95      (5.6%)   -1.6% ( -10% -    8%) 0.305
                            Term100      569.96      (4.2%)      561.59      (6.0%)   -1.5% ( -11% -    9%) 0.371
                          CountTerm     6833.17      (4.2%)     6736.75      (4.8%)   -1.4% ( -10% -    7%) 0.323
                  FilteredOrHighMed       52.38      (3.5%)       51.76      (2.2%)   -1.2% (  -6% -    4%) 0.200
                         OrHighRare      121.85      (3.7%)      120.51      (4.8%)   -1.1% (  -9% -    7%) 0.421
                             IntSet      397.61      (3.6%)      393.40      (3.6%)   -1.1% (  -7% -    6%) 0.349
                         DismaxTerm      624.07      (2.6%)      617.94      (2.6%)   -1.0% (  -6% -    4%) 0.232
                 CombinedAndHighMed       27.95      (4.2%)       27.69      (2.6%)   -0.9% (  -7% -    6%) 0.404
                         TermDTSort      184.68      (3.5%)      183.18      (2.3%)   -0.8% (  -6% -    5%) 0.385
                       SloppyPhrase        1.42      (2.9%)        1.41      (4.5%)   -0.8% (  -7% -    6%) 0.526
                 FilteredOrHighHigh       17.80      (1.6%)       17.67      (1.5%)   -0.8% (  -3% -    2%) 0.124
                  CombinedOrHighMed       27.47      (5.5%)       27.26      (4.4%)   -0.7% ( -10% -    9%) 0.641
                      TermMonthSort     2430.41      (3.5%)     2413.88      (3.0%)   -0.7% (  -6% -    6%) 0.513
                     FilteredPhrase       12.52      (2.0%)       12.43      (1.8%)   -0.7% (  -4% -    3%) 0.263
                      TermTitleSort       67.84      (4.2%)       67.41      (3.3%)   -0.6% (  -7% -    7%) 0.591
         FilteredOr2Terms2StopWords       65.16      (1.8%)       64.74      (1.3%)   -0.6% (  -3% -    2%) 0.210
                   FilteredOr3Terms       58.20      (2.3%)       57.84      (1.8%)   -0.6% (  -4% -    3%) 0.348
                   CountAndHighHigh       61.16      (1.6%)       60.79      (1.3%)   -0.6% (  -3% -    2%) 0.194
             CountFilteredOrHighMed       29.79      (1.7%)       29.62      (1.6%)   -0.6% (  -3% -    2%) 0.286
                  TermDayOfYearSort      341.54      (2.0%)      339.65      (1.7%)   -0.6% (  -4% -    3%) 0.343
                             Phrase        9.73      (2.9%)        9.68      (2.6%)   -0.5% (  -5% -    5%) 0.548
                            Respell       43.85      (3.0%)       43.64      (1.9%)   -0.5% (  -5% -    4%) 0.540
            CountFilteredOrHighHigh       25.38      (1.6%)       25.27      (1.4%)   -0.4% (  -3% -    2%) 0.357
                    DismaxOrHighMed       65.04      (2.3%)       64.80      (2.3%)   -0.4% (  -4% -    4%) 0.610
                CountFilteredIntNRQ       22.26      (1.7%)       22.18      (1.7%)   -0.3% (  -3% -    3%) 0.545
                             Fuzzy1       49.96      (3.5%)       49.80      (2.4%)   -0.3% (  -5% -    5%) 0.735
                       CombinedTerm       14.61      (3.7%)       14.57      (3.5%)   -0.3% (  -7% -    7%) 0.785
                             IntNRQ       48.09      (1.6%)       47.95      (1.3%)   -0.3% (  -3% -    2%) 0.501
                     FilteredIntNRQ       47.86      (1.6%)       47.71      (1.3%)   -0.3% (  -3% -    2%) 0.515
                     CountOrHighMed       95.08      (2.7%)       94.83      (1.5%)   -0.3% (  -4% -    4%) 0.706
                           Wildcard       58.21      (4.4%)       58.09      (3.8%)   -0.2% (  -8% -    8%) 0.875
                FilteredOrStopWords       10.96      (1.4%)       10.94      (1.9%)   -0.2% (  -3% -    3%) 0.703
                     FilteredOrMany        5.04      (1.6%)        5.03      (1.3%)   -0.2% (  -3% -    2%) 0.664
                CountFilteredPhrase       11.59      (1.4%)       11.57      (1.4%)   -0.2% (  -2% -    2%) 0.661
                             Fuzzy2       45.25      (3.3%)       45.17      (2.0%)   -0.2% (  -5% -    5%) 0.851
                CountFilteredOrMany        6.06      (1.2%)        6.05      (1.0%)   -0.2% (  -2% -    2%) 0.668
                    CountAndHighMed       92.80      (2.4%)       92.69      (2.1%)   -0.1% (  -4% -    4%) 0.870
                    CountOrHighHigh       63.38      (1.2%)       63.36      (1.2%)   -0.0% (  -2% -    2%) 0.940
                    AndHighOrMedMed       18.16      (2.7%)       18.16      (2.4%)   -0.0% (  -4% -    5%) 0.981
                FilteredAndHighHigh       14.92      (2.3%)       14.94      (2.9%)    0.1% (  -5% -    5%) 0.918
                            Prefix3       98.99      (3.8%)       99.07      (3.5%)    0.1% (  -6% -    7%) 0.941
                          OrHighMed       88.50      (2.6%)       88.59      (2.1%)    0.1% (  -4% -    4%) 0.889
                   DismaxOrHighHigh       46.10      (1.7%)       46.16      (2.8%)    0.1% (  -4% -    4%) 0.867
                  FilteredAnd3Terms      131.08      (1.5%)      131.28      (1.5%)    0.2% (  -2% -    3%) 0.746
               FilteredAndStopWords       11.78      (2.2%)       11.82      (2.8%)    0.4% (  -4% -    5%) 0.662
                CombinedAndHighHigh        7.20      (2.1%)        7.24      (1.9%)    0.5% (  -3% -    4%) 0.415
                    FilteredPrefix3       92.06      (3.9%)       92.61      (3.2%)    0.6% (  -6% -    8%) 0.601
                        CountOrMany        6.37      (2.0%)        6.41      (1.9%)    0.7% (  -3% -    4%) 0.264
                   AndMedOrHighHigh       20.85      (1.6%)       21.01      (1.3%)    0.7% (  -2% -    3%) 0.106
                 CombinedOrHighHigh        7.04      (4.5%)        7.10      (4.9%)    1.0% (  -8% -   10%) 0.520
                   IntervalsOrdered        3.00      (3.7%)        3.03      (3.9%)    1.1% (  -6% -    8%) 0.372
                        CountPhrase        3.28      (4.3%)        3.32      (3.1%)    1.1% (  -6% -    8%) 0.367
                         AndHighMed       67.26      (2.4%)       68.09      (1.4%)    1.2% (  -2% -    5%) 0.049
                           SpanNear        3.07      (4.9%)        3.11      (3.4%)    1.2% (  -6% -   10%) 0.350
                        AndHighHigh       26.61      (2.8%)       26.94      (2.8%)    1.3% (  -4% -    7%) 0.156
                 FilteredAndHighMed       43.51      (1.8%)       44.08      (1.9%)    1.3% (  -2% -    5%) 0.027
                         OrHighHigh       25.58      (2.2%)       25.99      (2.3%)    1.6% (  -2% -    6%) 0.024
        FilteredAnd2Terms2StopWords       73.91      (2.8%)       75.69      (1.4%)    2.4% (  -1% -    6%) 0.001
                 Or2Terms2StopWords       73.90      (3.3%)       76.12      (1.0%)    3.0% (  -1% -    7%) 0.000
                And2Terms2StopWords       71.57      (3.1%)       73.79      (1.3%)    3.1% (  -1% -    7%) 0.000
                          And3Terms       85.24      (3.7%)       88.75      (1.3%)    4.1% (   0% -    9%) 0.000
                        OrStopWords        9.89      (6.7%)       10.31      (5.6%)    4.2% (  -7% -   17%) 0.031
                           Or3Terms       77.31      (4.0%)       80.81      (1.5%)    4.5% (  -1% -   10%) 0.000
                             OrMany        5.53      (6.8%)        5.82      (3.0%)    5.1% (  -4% -   16%) 0.002
                       AndStopWords        9.17      (5.8%)        9.70      (3.5%)    5.7% (  -3% -   15%) 0.000
   WARNING: cat=Fuzzy1: hit counts differ: 11543+ vs 11524+
   WARNING: cat=Fuzzy2: hit counts differ: 12724+ vs 12716+
   WARNING: cat=Or2Terms2StopWords: hit counts differ: 14986+ vs 14985+
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3030054987

   The change from `ulp(double)` to `ulp(float)` seems to help decrease the p-value compared to previous benchmark result, it's great!
   <details>
   <summary> luceneutil </summary>
                               
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                             IntNRQ       48.58      (1.4%)       48.14      (2.0%)   -0.9% (  -4% -    2%) 0.090
                            Prefix3       98.75      (3.5%)       97.99      (4.3%)   -0.8% (  -8% -    7%) 0.534
                        CountOrMany        6.41      (2.2%)        6.36      (2.6%)   -0.7% (  -5% -    4%) 0.328
                      TermTitleSort       70.07      (5.0%)       69.57      (5.6%)   -0.7% ( -10% -   10%) 0.669
                        CountPhrase        3.28      (3.2%)        3.27      (3.5%)   -0.5% (  -6% -    6%) 0.616
                     FilteredIntNRQ       48.13      (1.6%)       47.88      (2.0%)   -0.5% (  -4% -    3%) 0.363
                    FilteredPrefix3       92.07      (3.5%)       91.63      (4.0%)   -0.5% (  -7% -    7%) 0.685
                          CountTerm     6724.13      (5.8%)     6693.36      (6.7%)   -0.5% ( -12% -   12%) 0.817
                CountFilteredOrMany        6.08      (1.6%)        6.05      (2.3%)   -0.4% (  -4% -    3%) 0.548
                     FilteredPhrase       12.62      (1.4%)       12.58      (2.1%)   -0.3% (  -3% -    3%) 0.554
                    CountOrHighHigh       63.73      (2.0%)       63.52      (2.5%)   -0.3% (  -4% -    4%) 0.647
            CountFilteredOrHighHigh       25.24      (1.3%)       25.18      (1.8%)   -0.3% (  -3% -    2%) 0.610
                   CountAndHighHigh       61.82      (1.5%)       61.66      (1.7%)   -0.3% (  -3% -    2%) 0.607
                CountFilteredIntNRQ       22.20      (2.1%)       22.15      (2.2%)   -0.2% (  -4% -    4%) 0.728
             CountFilteredOrHighMed       29.69      (1.4%)       29.62      (1.8%)   -0.2% (  -3% -    3%) 0.647
                       FilteredTerm       87.59      (5.9%)       87.40      (6.6%)   -0.2% ( -11% -   13%) 0.912
                            Respell       43.52      (2.0%)       43.45      (2.4%)   -0.2% (  -4% -    4%) 0.816
                  TermDayOfYearSort      344.16      (1.8%)      343.61      (1.2%)   -0.2% (  -3% -    2%) 0.745
                CountFilteredPhrase       11.50      (2.4%)       11.49      (2.1%)   -0.1% (  -4% -    4%) 0.866
                     CountOrHighMed       96.68      (2.5%)       96.57      (3.3%)   -0.1% (  -5% -    5%) 0.899
               FilteredAndStopWords       11.98      (4.2%)       11.97      (4.2%)   -0.1% (  -8% -    8%) 0.954
                 FilteredOrHighHigh       18.06      (3.3%)       18.05      (3.7%)   -0.1% (  -6% -    7%) 0.962
                FilteredAndHighHigh       15.22      (4.0%)       15.22      (3.9%)   -0.0% (  -7% -    8%) 0.971
                  FilteredOrHighMed       54.08      (4.6%)       54.06      (5.6%)   -0.0% (  -9% -   10%) 0.982
                       SloppyPhrase        1.44      (4.8%)        1.44      (3.9%)    0.0% (  -8% -    9%) 0.982
                    CountAndHighMed       93.74      (1.9%)       93.80      (3.4%)    0.1% (  -5% -    5%) 0.943
                           Wildcard       58.08      (3.5%)       58.13      (3.6%)    0.1% (  -6% -    7%) 0.937
                             Phrase        9.83      (2.0%)        9.85      (2.3%)    0.1% (  -4% -    4%) 0.839
                         DismaxTerm      634.49      (4.5%)      635.53      (6.3%)    0.2% ( -10% -   11%) 0.924
                   FilteredOr3Terms       59.59      (4.2%)       59.72      (5.2%)    0.2% (  -8% -   10%) 0.884
                             Fuzzy2       45.18      (3.2%)       45.29      (3.2%)    0.3% (  -6% -    6%) 0.802
         FilteredOr2Terms2StopWords       67.27      (6.2%)       67.45      (7.2%)    0.3% ( -12% -   14%) 0.901
                  FilteredAnd3Terms      131.54      (2.7%)      131.95      (3.3%)    0.3% (  -5% -    6%) 0.747
                  CombinedOrHighMed       28.71      (5.4%)       28.81      (5.7%)    0.4% ( -10% -   12%) 0.837
                      TermMonthSort     2460.88      (4.2%)     2470.76      (5.0%)    0.4% (  -8% -   10%) 0.784
                   AndMedOrHighHigh       20.92      (1.5%)       21.02      (2.1%)    0.5% (  -3% -    4%) 0.412
                 CombinedAndHighMed       29.00      (5.5%)       29.14      (5.6%)    0.5% ( -10% -   12%) 0.776
                         TermDTSort      190.27      (4.4%)      191.39      (5.1%)    0.6% (  -8% -   10%) 0.695
                            Term10K      584.68      (5.0%)      588.32      (6.6%)    0.6% ( -10% -   12%) 0.738
                    DismaxOrHighMed       66.04      (4.2%)       66.49      (4.3%)    0.7% (  -7% -    9%) 0.621
                     FilteredOrMany        5.09      (2.0%)        5.12      (3.0%)    0.7% (  -4% -    5%) 0.402
                             Fuzzy1       50.40      (4.2%)       50.76      (4.7%)    0.7% (  -7% -   10%) 0.613
                             Term1M      584.67      (4.9%)      589.05      (6.9%)    0.7% ( -10% -   13%) 0.694
                       CombinedTerm       14.72      (4.4%)       14.84      (4.6%)    0.8% (  -7% -   10%) 0.580
                    AndHighOrMedMed       18.29      (3.5%)       18.44      (2.2%)    0.8% (  -4% -    6%) 0.381
                          TermB1M1P      583.58      (5.0%)      588.67      (6.7%)    0.9% ( -10% -   13%) 0.643
                FilteredOrStopWords       10.94      (2.8%)       11.04      (3.3%)    0.9% (  -5% -    7%) 0.347
                CombinedAndHighHigh        7.19      (2.0%)        7.26      (2.9%)    0.9% (  -3% -    5%) 0.234
                            Term100      582.93      (5.0%)      588.80      (6.8%)    1.0% ( -10% -   13%) 0.594
                            TermB1M      583.60      (4.8%)      589.94      (6.6%)    1.1% (  -9% -   13%) 0.552
                 CombinedOrHighHigh        7.10      (3.1%)        7.17      (4.1%)    1.1% (  -5% -    8%) 0.341
                             IntSet      384.28      (3.5%)      388.53      (4.8%)    1.1% (  -6% -    9%) 0.404
                   IntervalsOrdered        2.93      (3.7%)        2.96      (3.4%)    1.1% (  -5% -    8%) 0.309
                           SpanNear        3.06      (4.8%)        3.09      (3.9%)    1.2% (  -7% -   10%) 0.397
                               Term      582.79      (5.0%)      589.71      (6.8%)    1.2% ( -10% -   13%) 0.528
                   DismaxOrHighHigh       45.65      (3.4%)       46.21      (3.1%)    1.2% (  -5% -    8%) 0.235
                         OrHighRare      117.32      (6.4%)      118.79      (7.2%)    1.2% ( -11% -   15%) 0.562
                          OrHighMed       90.35      (4.4%)       91.60      (4.7%)    1.4% (  -7% -   10%) 0.337
                         AndHighMed       68.62      (3.3%)       69.82      (3.9%)    1.7% (  -5% -    9%) 0.126
                 FilteredAndHighMed       44.21      (3.8%)       44.99      (3.5%)    1.8% (  -5% -    9%) 0.123
                        AndHighHigh       26.53      (2.8%)       27.08      (2.4%)    2.1% (  -3% -    7%) 0.012
                         OrHighHigh       25.43      (2.8%)       26.16      (2.4%)    2.8% (  -2% -    8%) 0.001
        FilteredAnd2Terms2StopWords       74.67      (6.7%)       77.28      (6.8%)    3.5% (  -9% -   18%) 0.101
                And2Terms2StopWords       73.37      (8.9%)       76.53      (9.4%)    4.3% ( -12% -   24%) 0.137
                 Or2Terms2StopWords       75.05      (8.1%)       78.52      (8.3%)    4.6% ( -10% -   22%) 0.075
                          And3Terms       85.24      (6.1%)       90.46      (4.3%)    6.1% (  -3% -   17%) 0.000
                           Or3Terms       77.07      (6.1%)       82.11      (3.9%)    6.5% (  -3% -   17%) 0.000
                             OrMany        5.44      (7.2%)        5.84      (3.9%)    7.3% (  -3% -   19%) 0.000
                        OrStopWords        9.49      (7.8%)       10.21      (4.0%)    7.6% (  -3% -   21%) 0.000
                       AndStopWords        8.90      (7.7%)        9.64      (3.0%)    8.4% (  -2% -   20%) 0.000
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-2993476199

   This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #14827:
URL: https://github.com/apache/lucene/pull/14827#discussion_r2180963012


##########
lucene/core/src/test/org/apache/lucene/search/TestScorerUtil.java:
##########
@@ -93,4 +94,20 @@ public void testLikelyImpactsEnum() throws IOException {
       }
     }
   }
+
+  public void testMinRequiredScore() {

Review Comment:
   Can you test many iterations at once? Something like:
   
   ```java
   int iters = atLeast(100);
   for (int iter = 0; iter < iters; ++iter) {
    // your test code
   }
   ```



##########
lucene/core/src/java/org/apache/lucene/search/ScorerUtil.java:
##########
@@ -116,6 +116,25 @@ public int length() {
     }
   }
 
+  /**
+   * Compute a minimum required score, so that (float) MathUtil.sumUpperBound(minRequiredScore +
+   * maxRemainingScore, numScorers) <= minCompetitiveScore. The computed value may not be the
+   * greatest value that meets this condition, which means that we may fail to filter out some docs.
+   * However, this doesn't hurt correctness, it just means that these docs will be filtered out
+   * later, and the extra work required to compute an optimal value would unlikely result in a
+   * speedup.
+   */
+  static double minRequiredScore(
+      double maxRemainingScore, float minCompetitiveScore, int numScorers) {
+    double minRequiredScore = minCompetitiveScore - maxRemainingScore;
+    double subtraction = Math.ulp((double) minCompetitiveScore);

Review Comment:
   ```suggestion
       double subtraction = Math.ulp(minCompetitiveScore); // note: we want the float ulp, not the double ulp
   ```
   
   Sorry, I realize that the current code was my suggestion, but looking at it again, we need to take the float ulp to make sure it converges quickly.



##########
lucene/core/src/test/org/apache/lucene/search/TestScorerUtil.java:
##########
@@ -93,4 +94,20 @@ public void testLikelyImpactsEnum() throws IOException {
       }
     }
   }
+
+  public void testMinRequiredScore() {
+    double maxRemainingScore = random().nextDouble();
+    float minCompetitiveScore = random().nextFloat();
+    int numScorers = random().nextInt(1, 10);
+
+    double minRequiredScore =
+        ScorerUtil.minRequiredScore(maxRemainingScore, minCompetitiveScore, numScorers);

Review Comment:
   I wonder if there's a way to track the number of iterations and fail the test if it took more than X (e.g. 10) iterations to converge.



##########
lucene/core/src/test/org/apache/lucene/search/TestScorerUtil.java:
##########
@@ -93,4 +94,20 @@ public void testLikelyImpactsEnum() throws IOException {
       }
     }
   }
+
+  public void testMinRequiredScore() {
+    double maxRemainingScore = random().nextDouble();
+    float minCompetitiveScore = random().nextFloat();
+    int numScorers = random().nextInt(1, 10);

Review Comment:
   Let's test with high numbers of clauses too?
   
   ```suggestion
       int numScorers = random().nextInt(1, 1000);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3018852667

   I played a bit more with your change, and it looks like we could make things even further as a follow-up through vectorization by using `(Int|Float)Vector#compress`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024631364

   This PR does not have an entry in lucene/CHANGES.txt. Consider adding one. If the PR doesn't need a changelog entry, then add the skip-changelog label to it and you will stop receiving this reminder on future updates to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3032803703

   I'm taking a look now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Specialize `filterCompetitiveHits` when have exact 2 clauses [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on PR #14827:
URL: https://github.com/apache/lucene/pull/14827#issuecomment-3024635431

   Sorry for the late reply. 
   I've digged a little bit into this problem, it seems the root cause is the initial `double` -> `float` cast, so I changed it from float to double, and the luceneutil do not complain about the hit count anymore. 
   
   The benchmark result under same setup is as follows:
   ```
                               TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                         OrHighRare      120.94      (2.7%)      122.37      (1.9%)    1.2% (  -3% -    5%) 0.109
                          OrHighMed       90.74      (3.6%)       92.40      (2.8%)    1.8% (  -4% -    8%) 0.073
                         AndHighMed       69.64      (2.8%)       71.32      (2.1%)    2.4% (  -2% -    7%) 0.002
                        AndHighHigh       28.49      (2.4%)       29.29      (2.1%)    2.8% (  -1% -    7%) 0.000
                         OrHighHigh       26.83      (2.5%)       27.60      (1.9%)    2.9% (  -1% -    7%) 0.000
   ```
   
   The while loop containing call to `Math.nextDown` doesn's seem to add much overhead. I locally added some simple print over the iteration count of the loop, they are almost entirely zero.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on code in PR #14827:
URL: https://github.com/apache/lucene/pull/14827#discussion_r2180136453


##########
lucene/core/src/java/org/apache/lucene/search/ScorerUtil.java:
##########
@@ -116,6 +116,25 @@ public int length() {
     }
   }
 
+  /**
+   * Compute a minimum required score, so that (float) MathUtil.sumUpperBound(minRequiredScore +
+   * maxRemainingScore, numScorers) <= minCompetitiveScore. The computed value may not be the
+   * greatest value that meets this condition, which means that we may fail to filter out some docs.
+   * However, this doesn't hurt correctness, it just means that these docs will be filtered out
+   * later, and the extra work required to compute an optimal value would unlikely result in a
+   * speedup.
+   */
+  static double minRequiredScore(
+      double maxRemainingScore, float minCompetitiveScore, int numScorers) {
+    double minRequiredScore = minCompetitiveScore - maxRemainingScore;
+    double subtraction = Math.ulp((double) minCompetitiveScore);
+    while ((float) MathUtil.sumUpperBound(minRequiredScore + maxRemainingScore, numScorers)
+        > minCompetitiveScore) {
+      minRequiredScore -= subtraction;
+    }
+    return minRequiredScore;
+  }

Review Comment:
   Thanks for extracting this function. Can you add unit tests?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Pre-calculate minRequiredScore to speedup filterCompetitiveHits [lucene]

Posted by "HUSTERGS (via GitHub)" <gi...@apache.org>.
HUSTERGS commented on code in PR #14827:
URL: https://github.com/apache/lucene/pull/14827#discussion_r2181744506


##########
lucene/core/src/test/org/apache/lucene/search/TestScorerUtil.java:
##########
@@ -93,4 +94,20 @@ public void testLikelyImpactsEnum() throws IOException {
       }
     }
   }
+
+  public void testMinRequiredScore() {
+    double maxRemainingScore = random().nextDouble();
+    float minCompetitiveScore = random().nextFloat();
+    int numScorers = random().nextInt(1, 10);
+
+    double minRequiredScore =
+        ScorerUtil.minRequiredScore(maxRemainingScore, minCompetitiveScore, numScorers);

Review Comment:
   Or should we assert it inside `minRequiredScore`, still a little bit ugly though



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org