You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "kaivalnp (via GitHub)" <gi...@apache.org> on 2023/12/12 07:15:02 UTC

[PR] Fix failing BaseVectorSimilarityQueryTestCase#testApproximate [lucene]

kaivalnp opened a new pull request, #12922:
URL: https://github.com/apache/lucene/pull/12922

   Discovered in #12921, and introduced in #12679 
   
   The first issue is that we weren't advancing the `VectorScorer` [here](https://github.com/apache/lucene/blob/cf13a9295052288b748ed8f279f05ee26f3bfd5f/lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java#L257-L262) -- so it was still un-positioned while trying to compute the similarity score
   
   Earlier in the PR, the underlying delegate of the `FilteredDocIdSetIterator` was `scorer.iterator()` (see [here](https://github.com/apache/lucene/blob/cad565439be512ac6e95a698007b1fc971173f00/lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java#L107)) -- so we didn't need to explicitly advance it
   
   Later, we decided to maintain parity to `AbstractKnnVectorQuery` and introduce filtering in `AbstractVectorSimilarityQuery` (see [this commit](https://github.com/apache/lucene/commit/5096790f281e477c529a7c8311aeb353ccdffdeb)) to determine the `visitLimit` of approximate search -- after which the underlying iterator changed to the accepted docs (see [here](https://github.com/apache/lucene/blob/5096790f281e477c529a7c8311aeb353ccdffdeb/lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java#L255)) and I missed advancing the `VectorScorer` explicitly..
   
   After doing so, we no longer get the original `java.lang.ArrayIndexOutOfBoundsException` -- but the `BaseVectorSimilarityQueryTestCase#testApproximate` starts failing because it falls back to exact search, as the limit of the prefilter is met during graph search
   
   Relaxed the parameters of the test to fix this (making the filter less restrictive, and trying to visit a fewer number of nodes so that approximate search completes without hitting its limit)
   
   Sorry for missing this earlier!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Fix failing BaseVectorSimilarityQueryTestCase#testApproximate [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #12922:
URL: https://github.com/apache/lucene/pull/12922#discussion_r1424385137


##########
lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java:
##########
@@ -255,6 +255,11 @@ static VectorSimilarityScorer fromAcceptDocs(
           new FilteredDocIdSetIterator(acceptDocs) {
             @Override
             protected boolean match(int doc) throws IOException {
+              // Advance the scorer
+              if (!scorer.advanceExact(doc)) {
+                return false;
+              }
+

Review Comment:
   I added a commit to demonstrate this^, please let me know if we can improve this / do it some other way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Fix failing BaseVectorSimilarityQueryTestCase#testApproximate [lucene]

Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on code in PR #12922:
URL: https://github.com/apache/lucene/pull/12922#discussion_r1423943933


##########
lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java:
##########
@@ -255,6 +255,11 @@ static VectorSimilarityScorer fromAcceptDocs(
           new FilteredDocIdSetIterator(acceptDocs) {
             @Override
             protected boolean match(int doc) throws IOException {
+              // Advance the scorer
+              if (!scorer.advanceExact(doc)) {
+                return false;
+              }
+

Review Comment:
   This tells me that none of the filtered cases drop to using exact search? If they did they would match on docs that were actually filtered out. 
   
   Could you add a test exercising this path?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Fix failing BaseVectorSimilarityQueryTestCase#testApproximate [lucene]

Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on code in PR #12922:
URL: https://github.com/apache/lucene/pull/12922#discussion_r1424532683


##########
lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java:
##########
@@ -255,6 +255,11 @@ static VectorSimilarityScorer fromAcceptDocs(
           new FilteredDocIdSetIterator(acceptDocs) {
             @Override
             protected boolean match(int doc) throws IOException {
+              // Advance the scorer
+              if (!scorer.advanceExact(doc)) {
+                return false;
+              }
+

Review Comment:
   adding an assert is nice :). In reality it would throw trying to read garbage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Fix failing BaseVectorSimilarityQueryTestCase#testApproximate [lucene]

Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent merged PR #12922:
URL: https://github.com/apache/lucene/pull/12922


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Fix failing BaseVectorSimilarityQueryTestCase#testApproximate [lucene]

Posted by "kaivalnp (via GitHub)" <gi...@apache.org>.
kaivalnp commented on code in PR #12922:
URL: https://github.com/apache/lucene/pull/12922#discussion_r1424377622


##########
lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java:
##########
@@ -255,6 +255,11 @@ static VectorSimilarityScorer fromAcceptDocs(
           new FilteredDocIdSetIterator(acceptDocs) {
             @Override
             protected boolean match(int doc) throws IOException {
+              // Advance the scorer
+              if (!scorer.advanceExact(doc)) {
+                return false;
+              }
+

Review Comment:
   > This tells me that none of the filtered cases drop to using exact search?
   
   We do have a test case that covers this: [`testRandomFilter`](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/test/org/apache/lucene/search/BaseVectorSimilarityQueryTestCase.java#L158) randomly [filters a sub-range](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/test/org/apache/lucene/search/BaseVectorSimilarityQueryTestCase.java#L160-L161) of documents and [expects all of them to be found](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/test/org/apache/lucene/search/BaseVectorSimilarityQueryTestCase.java#L183-L184) -- which always falls back to exact search (because a `traversalSimilarity` of `Float.NEGATIVE_INFINITY` [here](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/test/org/apache/lucene/search/BaseVectorSimilarityQueryTestCase.java#L172) will visit the entire graph and exhaust any limit
 )
   
   While running this test, I put debug points [here](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/java/org/apache/lucene/search/AbstractVectorSimilarityQuery.java#L259) to check the `scorer`, and it was always unpositioned (`docid` = -1) but doesn't throw an error in *many* cases
   
   The error is thrown when the underlying [`values`](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/java/org/apache/lucene/search/VectorScorer.java#L96) are randomly chosen to `SimpleTextFloatVectorValues` - try `./gradlew test --tests TestFloatVectorSimilarityQuery.testRandomFilter -Dtests.seed=119135B1F0803918` for a failing case (as mentioned [here](https://github.com/apache/lucene/pull/12679#issuecomment-1851062374), thanks @epotyom!)
   
   In other cases (for example when `values` are `DenseOffHeapVectorValues`), the scorer *does not fail even when unpositioned* and returns some (garbage) value - but the count of results is still correct (and [asserted](https://github.com/apache/lucene/blob/e18f9b1eb04a70d3fe5d8431a3d6724f187f050c/lucene/core/src/test/org/apache/lucene/search/BaseVectorSimilarityQueryTestCase.java#L180-L184))
   
   I wonder if we can put an `assert values.docID() != -1` before [this](https://github.com/apache/lucene/blob/1ac1b1cadc66364c8baca48e8334aa2855cd18b6/lucene/core/src/java/org/apache/lucene/search/VectorScorer.java#L90) and [this](https://github.com/apache/lucene/blob/1ac1b1cadc66364c8baca48e8334aa2855cd18b6/lucene/core/src/java/org/apache/lucene/search/VectorScorer.java#L120) line to ensure it is positioned before trying to compute scores? -- I tried this offline, and the issue was caught immediately



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org