You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2021/08/15 06:00:18 UTC
[GitHub] [lucene] zacharymorn commented on a change in pull request #240: LUCENE-10002: Deprecate IndexSearch#search(Query, Collector) in favor of IndexSearcher#search(Query, CollectorManager)
zacharymorn commented on a change in pull request #240:
URL: https://github.com/apache/lucene/pull/240#discussion_r689035631
##########
File path: lucene/core/src/java/org/apache/lucene/search/TopFieldCollectorManager.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * Create a TopFieldCollectorManager which uses a shared hit counter to maintain number of hits and
+ * a shared {@link MaxScoreAccumulator} to propagate the minimum score across segments if the
+ * primary sort is by relevancy.
+ *
+ * <p>Note that a new collectorManager should be created for each search due to its internal states.
+ */
+public class TopFieldCollectorManager implements CollectorManager<TopFieldCollector, TopFieldDocs> {
+ private final Sort sort;
+ private final int numHits;
+ private final FieldDoc after;
+ private final HitsThresholdChecker hitsThresholdChecker;
+ private final MaxScoreAccumulator minScoreAcc;
+ private final List<TopFieldCollector> collectors;
+
+ public TopFieldCollectorManager(Sort sort, int numHits, FieldDoc after, int totalHitsThreshold) {
+ if (totalHitsThreshold < 0) {
+ throw new IllegalArgumentException(
+ "totalHitsThreshold must be >= 0, got " + totalHitsThreshold);
+ }
+
+ this.sort = sort;
+ this.numHits = numHits;
+ this.after = after;
+ /*
+ nocommit
+ Should the following two be passed in instead? Possible custom initialization based on executor status and slices?
+ On the other hand, in a single-threaded environment, shared HitsThresholdChecker and MaxScoreAccumulator should be fast without lock contention anyway?
Review comment:
Thanks for the great suggestion! I just ran luceneutil, and it turns out that the changes do get slower due to this thread-safe `HitsThresholdChecker` (confirmed by playing with the atomic long inside a bit). Here is the full benchmark result:
```
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value
HighTermTitleBDVSort 52.86 (14.0%) 42.72 (6.6%) -19.2% ( -34% - 1%) 0.000
HighTermMonthSort 31.66 (10.7%) 25.90 (6.3%) -18.2% ( -31% - -1%) 0.000
HighTermDayOfYearSort 24.00 (15.7%) 20.02 (8.5%) -16.6% ( -35% - 9%) 0.000
TermDTSort 20.22 (9.0%) 17.13 (6.6%) -15.3% ( -28% - 0%) 0.000
MedTerm 1270.02 (3.9%) 1208.29 (5.1%) -4.9% ( -13% - 4%) 0.001
HighTerm 941.14 (4.0%) 900.05 (5.9%) -4.4% ( -13% - 5%) 0.006
OrHighNotMed 531.63 (5.6%) 514.68 (6.2%) -3.2% ( -14% - 9%) 0.089
LowTerm 1112.50 (5.1%) 1081.48 (6.2%) -2.8% ( -13% - 8%) 0.118
MedSpanNear 12.50 (2.2%) 12.18 (2.2%) -2.5% ( -6% - 1%) 0.000
HighSpanNear 12.82 (1.7%) 12.52 (1.7%) -2.4% ( -5% - 1%) 0.000
OrHighNotLow 562.50 (6.9%) 550.15 (5.6%) -2.2% ( -13% - 11%) 0.269
OrHighNotHigh 577.10 (5.6%) 565.51 (6.7%) -2.0% ( -13% - 10%) 0.303
OrNotHighHigh 633.91 (6.7%) 621.85 (6.0%) -1.9% ( -13% - 11%) 0.345
HighIntervalsOrdered 6.94 (3.1%) 6.81 (3.2%) -1.9% ( -7% - 4%) 0.061
PKLookup 174.06 (4.2%) 171.17 (4.5%) -1.7% ( -9% - 7%) 0.230
OrHighLow 283.56 (5.3%) 279.05 (5.8%) -1.6% ( -12% - 10%) 0.365
MedIntervalsOrdered 19.00 (3.7%) 18.74 (3.6%) -1.4% ( -8% - 6%) 0.237
Fuzzy1 59.38 (11.6%) 58.65 (12.0%) -1.2% ( -22% - 25%) 0.740
HighSloppyPhrase 12.68 (2.7%) 12.53 (2.1%) -1.1% ( -5% - 3%) 0.135
Fuzzy2 63.95 (8.6%) 63.40 (9.0%) -0.9% ( -17% - 18%) 0.759
MedSloppyPhrase 17.53 (2.7%) 17.38 (2.2%) -0.8% ( -5% - 4%) 0.279
LowSloppyPhrase 23.90 (3.0%) 23.70 (2.8%) -0.8% ( -6% - 5%) 0.354
Respell 39.23 (3.1%) 38.90 (3.1%) -0.8% ( -6% - 5%) 0.403
AndHighMed 61.17 (3.8%) 60.72 (3.8%) -0.7% ( -8% - 7%) 0.542
OrHighMed 48.81 (3.0%) 48.46 (3.0%) -0.7% ( -6% - 5%) 0.444
LowIntervalsOrdered 14.64 (2.3%) 14.55 (2.3%) -0.6% ( -5% - 4%) 0.380
LowSpanNear 11.65 (1.6%) 11.57 (1.6%) -0.6% ( -3% - 2%) 0.223
AndHighHigh 16.57 (3.6%) 16.46 (4.0%) -0.6% ( -7% - 7%) 0.608
IntNRQ 17.25 (28.0%) 17.17 (27.3%) -0.5% ( -43% - 76%) 0.959
OrHighHigh 9.28 (2.5%) 9.24 (2.5%) -0.4% ( -5% - 4%) 0.635
MedPhrase 72.99 (3.3%) 72.75 (3.7%) -0.3% ( -7% - 6%) 0.763
LowPhrase 78.02 (3.5%) 77.77 (4.0%) -0.3% ( -7% - 7%) 0.786
BrowseMonthSSDVFacets 3.19 (1.8%) 3.19 (1.7%) -0.2% ( -3% - 3%) 0.653
BrowseDayOfYearSSDVFacets 3.01 (1.5%) 3.01 (1.2%) -0.0% ( -2% - 2%) 0.991
Wildcard 27.98 (2.8%) 27.98 (2.7%) 0.0% ( -5% - 5%) 0.981
AndHighLow 454.42 (5.4%) 455.10 (4.6%) 0.1% ( -9% - 10%) 0.925
HighPhrase 289.77 (5.1%) 290.54 (4.0%) 0.3% ( -8% - 9%) 0.854
OrNotHighLow 521.43 (6.5%) 523.38 (5.7%) 0.4% ( -11% - 13%) 0.846
Prefix3 39.82 (4.8%) 40.02 (3.1%) 0.5% ( -7% - 8%) 0.707
OrNotHighMed 560.49 (5.8%) 563.51 (5.5%) 0.5% ( -10% - 12%) 0.762
BrowseMonthTaxoFacets 0.99 (4.8%) 1.00 (4.6%) 0.6% ( -8% - 10%) 0.702
BrowseDateTaxoFacets 0.94 (4.6%) 0.94 (4.6%) 0.6% ( -8% - 10%) 0.683
BrowseDayOfYearTaxoFacets 0.94 (4.5%) 0.94 (4.5%) 0.7% ( -7% - 10%) 0.638
```
To fix the performance regression, but still try to keep the logic simple then, I'm considering adding an additional flag `supportsConcurrency` to the collector managers' constructor like such:
```
public TopScoreDocCollectorManager(int numHits, ScoreDoc after, int totalHitsThreshold, boolean supportsConcurrency) {
...
this.hitsThresholdChecker = supportsConcurrency ?
HitsThresholdChecker.createShared(Math.max(totalHitsThreshold, numHits)) :
HitsThresholdChecker.create(Math.max(totalHitsThreshold, numHits));
this.minScoreAcc = supportsConcurrency ? new MaxScoreAccumulator() : null;
}
public TopFieldCollectorManager(Sort sort, int numHits, FieldDoc after, int totalHitsThreshold, boolean supportsConcurrency) {
...
this.hitsThresholdChecker = supportsConcurrency ?
HitsThresholdChecker.createShared(Math.max(totalHitsThreshold, numHits)) :
HitsThresholdChecker.create(Math.max(totalHitsThreshold, numHits));
this.minScoreAcc = supportsConcurrency ? new MaxScoreAccumulator() : null;
}
```
What do you think about this approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org