You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "benwtrent (via GitHub)" <gi...@apache.org> on 2025/10/10 15:59:42 UTC
[I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
benwtrent opened a new issue, #15324:
URL: https://github.com/apache/lucene/issues/15324
### Description
We are hitting a weird EOF in Lucene. It appears that its possible for an essentialQueue scorer can hit `NO_MORE_DOCS`, advance the TermScorer past its maxDoc, and then attempt to gather a score.
I do see we adjusted this path in 10.2: https://github.com/apache/lucene/pull/14186
Haven't been able to test this same data in 10.2 yet.
But, I have been staring at these code paths for days and just cannot see how we are progressing the `top` iterator to a doc past the maxDoc in the segment when using a filter.
Note:
- It HAS to be a filter. I tried the same exact query, but with a `must` clause with boosting by `0` (so not contributing to score at all), and we don't hit the EOF
- It MUST be done with disjunctions, I tried with conjunctions and a filter, and it worked just fine
- It requires a fairly restricted filter (matching a few percentage of docs or less than a percent of docs).
- One of the clauses much match more docs than the other (the particular failure is when one clause matches 3x more docs). But both match less than a 1/3 of the docs. But are less restrictive than the filter.
As for the TermScorer and such, that code path hasn't changed in a long time. basically, it never verifies its at NO_MORE_DOCS when scoring, it just always accepts the "advanceExact" as `true` even when that passes the maxDoc. Which seems like a mistake already.
I would expect `advanceExact` to return `false` if its advancing past maxDoc right?
```
org.apache.lucene.store.MemorySegmentIndexInput$SingleSegmentImpl.readByte(MemorySegmentIndexInput.java:762)
at org.apache.lucene.codecs.lucene90.Lucene90NormsProducer$3.longValue(Lucene90NormsProducer.java:399)
at org.apache.lucene.search.TermScorer.score(TermScorer.java:93)
at org.apache.lucene.search.DisjunctionSumScorer.score(DisjunctionSumScorer.java:43)
at org.apache.lucene.search.DisjunctionScorer.score(DisjunctionScorer.java:176)
at org.apache.lucene.search.MaxScoreBulkScorer.scoreInnerWindowWithFilter(MaxScoreBulkScorer.java:201)
at org.apache.lucene.search.MaxScoreBulkScorer.scoreInnerWindow(MaxScoreBulkScorer.java:147)
at org.apache.lucene.search.MaxScoreBulkScorer.score(MaxScoreBulkScorer.java:128)
at org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
at org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:461)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:810)
at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:388)
at org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:368)
at java.util.concurrent.FutureTask.run(FutureTask.java:328)
at org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
at org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
```
### Version and environment details
Lucene 10.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3393615943
> I would expect advanceExact to return false if its advancing past maxDoc right?
The contract says that this behavior is undefined. The initial use-case for `advanceExact` was speeding up faceting, so it would be called on doc IDs passed to `LeafCollector#collect`, which should always be in the [0, maxDoc) range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3395230859
I reread the advanceExact contract, for sure, the caller needs to be smart enough to provide only valid values.
That leaves the issue, why is this path failing :(. Still investigating to figure that one out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent closed issue #15324: Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter
URL: https://github.com/apache/lucene/issues/15324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3458254028
With some more digging, what we found is that `DisjunctionScorer` will utilize a cached view of its `TwoPhase.verifiedMatches` on a later document. This will cause the `getSubMatches` to be completely wrong, matching a `NO_MORE_DOCS` value.
I am not sure why this is the case, `TwoPhaseIterator.asDocIdSetIterator(twoPhase);` should be calling `matches()` on every doc. But there apparently is a case where this doesn't happen.
Whatever the cause, `approximation` and `TwoPhase#verifiedMatches` get out of sync in `DisjunctionScorer` when used in `MaxScoreBulkScorer` (the underlying scorer in this case was also a PhraseScorer).
Still trying to figure out why these get out of sync :/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "pmpailis (via GitHub)" <gi...@apache.org>.
pmpailis commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3443273632
A few more observations on the issue:
* We seem to also trip the assertion in `Lucene101PostingsReader.BlockPostingsEnum#accumulatePendingPositions`
```java
assert posPendingCount > 0;
```
when handling `UNARY` encoding.
* The original error originates when `Lucene101PostingsReader.BlockPostingsEnum#docID` returns `NO_MORE_DOCS`. Tracking this back, it seems that we try to access the `NO_MORE_DOCS` entry in `Lucene101PostingsReader.BlockPostingsEnum#docBuffer`, which AFAICT is set in `Lucene101PostingsReader.BlockPostingsEnum#refillRemainder` . Seems like we have exhausted the iterator, yet still we try to access and load postings.
Still looking into this, whilst also trying to generate a reproducible test scenario.
Also, FWIW this is also reproducible in 10.2.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3407780651
Updated to 10.2 with the same segment geometry and queries, and still getting the EOF. When we do leaf partitions and indicate we want to search the entire leaf, that generally means the `max` value we initially provide to `MaxScoreBulkScorer` is `NO_MORE_DOCS`
```
public static LeafReaderContextPartition createForEntireSegment(LeafReaderContext ctx) {
return new LeafReaderContextPartition(
ctx, 0, DocIdSetIterator.NO_MORE_DOCS, ctx.reader().maxDoc());
}
```
within `scoreInnerWindowWithFilter` it seems like:
```
int innerWindowMax = MathUtil.unsignedMin(max, innerWindowMin + INNER_WINDOW_SIZE);
```
Could end up being a value between the actual max doc in the segment and `NO_MORE_DOCS`, unless we force that `max` to be the the iterator max, but it seems the "default" behavior is for the max passed in to be `outerWindowMax` which could be `NO_MORE_DOCS`.
I am still not 100% sure how this ends up causing `top.doc == top.maxDoc`, but we end up in that state some how...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3411689031
The more I read this method, the more I think the bug might be in impacts. I noticed that we don't get an EOF when gathering all hits (instead of doing TOP_SCORES).
I wonder if we are hitting some funky edge case in impacts iteration where it caches the doc id or iterates past its max doc some how.
Still trying to this replicated, its pretty tricky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Strange EOF in 10.1.0 with MaxScoreBulkScorer#scoreInnerWindowWithFilter [lucene]
Posted by "john-wagster (via GitHub)" <gi...@apache.org>.
john-wagster commented on issue #15324:
URL: https://github.com/apache/lucene/issues/15324#issuecomment-3443324970
> Also, FWIW this is also reproducible in 10.2.2.
10.3.1 as well (es main). I'm actively working on getting a locally reproducible lucene test case now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org