You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ro...@apache.org on 2022/12/01 12:32:09 UTC
[lucene] branch branch_9x updated (3b497145b91 -> 07866ec2674)
This is an automated email from the ASF dual-hosted git repository.
romseygeek pushed a change to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
from 3b497145b91 ExitableTerms to override getMin and getMax (#11985)
new 8e4a4a0e7a6 Give OffsetsRetrievalStrategy implementations public constructors (#11983)
new 07866ec2674 Don't let merged passages push out lower-scoring ones (#11990)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
lucene/CHANGES.txt | 7 +++++++
.../search/matchhighlight/OffsetsFromMatchIterator.java | 2 +-
.../lucene/search/matchhighlight/OffsetsFromPositions.java | 2 +-
.../apache/lucene/search/matchhighlight/PassageSelector.java | 7 ++++++-
.../lucene/search/matchhighlight/TestPassageSelector.java | 12 ++++++++++++
5 files changed, 27 insertions(+), 3 deletions(-)
[lucene] 02/02: Don't let merged passages push out lower-scoring ones (#11990)
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
romseygeek pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 07866ec26741dc03af225591c8006d613b8822dd
Author: Alan Woodward <ro...@apache.org>
AuthorDate: Thu Dec 1 12:25:29 2022 +0000
Don't let merged passages push out lower-scoring ones (#11990)
PassageScorer uses a priority queue of size maxPassages to keep track of
which highlighted passages are worth returning to the user. Once all
passages have been collected, we go through and merge overlapping
passages together, but this reduction in the number of passages is not
compensated for by re-adding the highest-scoring passages that were pushed
out of the queue by passages which have been merged away.
This commit increases the size of the priority queue to try and account for
overlapping passages that will subsequently be merged together.
---
lucene/CHANGES.txt | 4 ++++
.../apache/lucene/search/matchhighlight/PassageSelector.java | 7 ++++++-
.../lucene/search/matchhighlight/TestPassageSelector.java | 12 ++++++++++++
3 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 1ce3aef7994..a22f9e87366 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -72,6 +72,10 @@ Bug Fixes
* GITHUB#11950: Fix NPE in BinaryRangeFieldRangeQuery variants when the queried field doesn't exist
in a segment or is of the wrong type. (Greg Miller)
+* GITHUB#11990: PassageSelector now has a larger minimum size for its priority queue,
+ so that subsequent passage merges don't mean that we return too few passages in
+ total. (Alan Woodward, Dawid Weiss)
+
Optimizations
---------------------
* GITHUB#11738: Optimize MultiTermQueryConstantScoreWrapper when a term is present that matches all
diff --git a/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/PassageSelector.java b/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/PassageSelector.java
index cdb07d9e611..05cba2f1b1e 100644
--- a/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/PassageSelector.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/PassageSelector.java
@@ -88,9 +88,13 @@ public class PassageSelector {
return Collections.emptyList();
}
+ // minimum priority queue size of 16 so that small maxPassages values don't
+ // return too few passages due to subsequent passage merges
+ int pqSize = Math.max(16, maxPassages);
+
// Best passages so far.
PriorityQueue<Passage> pq =
- new PriorityQueue<>(maxPassages) {
+ new PriorityQueue<>(pqSize) {
@Override
protected boolean lessThan(Passage a, Passage b) {
return passageScorer.compare(a, b) < 0;
@@ -215,6 +219,7 @@ public class PassageSelector {
}
// Remove nullified slots.
+ last = Math.min(last, maxPassages);
if (passages.length != last) {
passages = ArrayUtil.copyOfSubArray(passages, 0, last);
}
diff --git a/lucene/highlighter/src/test/org/apache/lucene/search/matchhighlight/TestPassageSelector.java b/lucene/highlighter/src/test/org/apache/lucene/search/matchhighlight/TestPassageSelector.java
index 95fa8730991..55e58a86f94 100644
--- a/lucene/highlighter/src/test/org/apache/lucene/search/matchhighlight/TestPassageSelector.java
+++ b/lucene/highlighter/src/test/org/apache/lucene/search/matchhighlight/TestPassageSelector.java
@@ -103,6 +103,18 @@ public class TestPassageSelector extends LuceneTestCase {
checkPassages("...>123456<...", value, 6, 1, new OffsetRange(1, value.length()));
}
+ @Test
+ public void overlappingResultsDontPushOutLowerScoringResults() {
+ String value =
+ "a fair amount of space and then two matches here and then two big long spaces for matches";
+ checkPassages(
+ "...and then >two< >matches<...|...and then >two< big lon...",
+ value,
+ 20,
+ 2,
+ ranges(new OffsetRange(32, 35), new OffsetRange(36, 43), new OffsetRange(58, 61)));
+ }
+
@Test
public void highlightLargerThanWindowWithSubranges() {
String value = "0123456789a";
[lucene] 01/02: Give OffsetsRetrievalStrategy implementations public constructors (#11983)
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
romseygeek pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 8e4a4a0e7a60fb29778168f8e59e076a66ec0146
Author: Alan Woodward <ro...@apache.org>
AuthorDate: Mon Nov 28 16:22:46 2022 +0000
Give OffsetsRetrievalStrategy implementations public constructors (#11983)
OffsetsFromMatchIterator and OffsetsFromPositions both have package-
private constructors, which makes them difficult to use as components in a
separate highlighter implementation.
---
lucene/CHANGES.txt | 3 +++
.../apache/lucene/search/matchhighlight/OffsetsFromMatchIterator.java | 2 +-
.../org/apache/lucene/search/matchhighlight/OffsetsFromPositions.java | 2 +-
3 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 2bfd8190f9e..1ce3aef7994 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -115,6 +115,9 @@ Other
* GITHUB#11752: Added interface to relate a LatLonShape with another shape represented as Component2D. (Navneet Verma)
+* GITHUB#11983: Make constructors for OffsetFromPositions and OffsetsFromMatchIterator
+ public. (Alan Woodward)
+
Build
---------------------
diff --git a/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromMatchIterator.java b/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromMatchIterator.java
index 17104a01a80..d7c42ebb03e 100644
--- a/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromMatchIterator.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromMatchIterator.java
@@ -30,7 +30,7 @@ public final class OffsetsFromMatchIterator implements OffsetsRetrievalStrategy
private final String field;
private final OffsetsFromPositions noOffsetsFallback;
- OffsetsFromMatchIterator(String field, OffsetsFromPositions noOffsetsFallback) {
+ public OffsetsFromMatchIterator(String field, OffsetsFromPositions noOffsetsFallback) {
this.field = field;
this.noOffsetsFallback = Objects.requireNonNull(noOffsetsFallback);
}
diff --git a/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromPositions.java b/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromPositions.java
index e460541c115..20a67194299 100644
--- a/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromPositions.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/matchhighlight/OffsetsFromPositions.java
@@ -36,7 +36,7 @@ public final class OffsetsFromPositions implements OffsetsRetrievalStrategy {
private final String field;
private final Analyzer analyzer;
- OffsetsFromPositions(String field, Analyzer analyzer) {
+ public OffsetsFromPositions(String field, Analyzer analyzer) {
this.field = field;
this.analyzer = analyzer;
}