You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "sjs004 (via GitHub)" <gi...@apache.org> on 2026/02/05 19:07:08 UTC
[I] WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations [lucene]
sjs004 opened a new issue, #15668:
URL: https://github.com/apache/lucene/issues/15668
### Description
The `PlainHighlighter` via [WeightedSpanTermExtractor.java#L447](https://github.com/apache/lucene/blob/releases/lucene/10.3.2/lucene/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java#L447) is prone to crashing when rewriting modern Lucene query types. This happens because the internal `DelegatingLeafReader` used for highlighting throws `UnsupportedOperationException` when `getFieldInfos()` is called.
**The Recurrent Issue**
As Lucene adds more optimizations to query rewrite() methods (e.g., checking index statistics or DocValues existence), these queries begin to fail when processed by the highlighter.
This pattern was previously observed with `FieldExistsQuery` (fixed in https://github.com/apache/lucene/pull/12088 by explicitly ignoring it). I am now observing the same crash in [OpenSearch](https://github.com/opensearch-project/OpenSearch/issues/20496) with `SortedNumericDocValuesRangeQuery` (via IndexOrDocValuesQuery), which attempts to access DocValuesSkipper and subsequently getFieldInfos() during rewrite
This creates a "whack-a-mole" situation where every new query optimization potentially breaks the highlighter.
**Example Failure**
When highlighting a boolean query containing a range filter:
```
Caused by: java.lang.UnsupportedOperationException
at org.apache.lucene.search.highlight.WeightedSpanTermExtractor$DelegatingLeafReader.getFieldInfos(WeightedSpanTermExtractor.java:447)
at org.apache.lucene.index.DocValuesSkipper.globalMinValue(DocValuesSkipper.java:137)
at org.apache.lucene.document.SortedNumericDocValuesRangeQuery.rewrite(SortedNumericDocValuesRangeQuery.java:101)
```
**Proposal & Discussion**
I believe we should address this structurally rather than adding exceptions for every new query type.
**Option 1**: The Structural Fix - Modify DelegatingLeafReader.getFieldInfos() to return FieldInfos instead of throwing an exception
PS: I am not very familiar with lucene codebase & not so sure if this option is feasible or it may require a long time to fix
**Option 2**: The Targeted Fix - Continue the pattern established in [PR #12088](https://github.com/apache/lucene/pull/12088) by adding IndexOrDocValuesQuery (and others) to the "ignore" list in `WeightedSpanTermExtractor.extract()` method
I am happy to submit a PR for option 2 & maybe option 1 as well if team suggests some solution
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations [lucene]
Posted by "sjs004 (via GitHub)" <gi...@apache.org>.
sjs004 commented on issue #15668:
URL: https://github.com/apache/lucene/issues/15668#issuecomment-3860287951
Thanks @romseygeek & @jainankitk . I will test with changes suggested by @romseygeek & raise a PR if it works without breaking any tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations [lucene]
Posted by "jainankitk (via GitHub)" <gi...@apache.org>.
jainankitk commented on issue #15668:
URL: https://github.com/apache/lucene/issues/15668#issuecomment-3858486473
Thanks @sjs004 for filing this issue. Seems like a bug indeed. While option 1 seems right approach to avoid this regression for new query types, I don't understand this code path very well. Though, I am happy to review PR for other approach if you are planning to raise one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations [lucene]
Posted by "sjs004 (via GitHub)" <gi...@apache.org>.
sjs004 commented on issue #15668:
URL: https://github.com/apache/lucene/issues/15668#issuecomment-3867740156
Hi @jainankitk @romseygeek
I have raised the PR: https://github.com/apache/lucene/pull/15678
Please review
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek closed issue #15668: WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations
URL: https://github.com/apache/lucene/issues/15668
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] WeightedSpanTermExtractor is fragile to modern query rewrites due to DelegatingLeafReader limitations [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on issue #15668:
URL: https://github.com/apache/lucene/issues/15668#issuecomment-3859990103
I had a quick look, and I think a change to `WeightedSpanTermExtractor` that looks something like this should work:
```
diff --git a/lucene/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java b/lucene/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java
index 1370afd736b..a16fff06467 100644
--- a/lucene/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java
+++ b/lucene/highlighter/src/java/org/apache/lucene/search/highlight/WeightedSpanTermExtractor.java
@@ -27,6 +27,7 @@ import java.util.Set;
import org.apache.lucene.analysis.CachingTokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.FieldInfo;
import org.apache.lucene.index.FieldInfos;
import org.apache.lucene.index.FilterLeafReader;
import org.apache.lucene.index.IndexReader;
@@ -444,7 +445,12 @@ public class WeightedSpanTermExtractor {
@Override
public FieldInfos getFieldInfos() {
- throw new UnsupportedOperationException(); // TODO merge them
+ return new FieldInfos(new FieldInfo[]{ super.getFieldInfos().fieldInfo(DelegatingLeafReader.FIELD_NAME) }) {
+ @Override
+ public FieldInfo fieldInfo(String fieldName) {
+ return super.fieldInfo(DelegatingLeafReader.FIELD_NAME);
+ }
+ };
}
@Override
```
Feel free to take this and run with it @sjs004 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org