You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "gsmiller (via GitHub)" <gi...@apache.org> on 2024/08/27 23:16:56 UTC

[PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

gsmiller opened a new pull request, #13693:
URL: https://github.com/apache/lucene/pull/13693

   ### Description
   
   This is a small optimization that treats "slow" multi-term queries (e.g., TermInSet, RegexpQuery, etc.) as ordinal ranges when the query terms create a contiguous range. This is likely a rare case, but it doesn't seem too hard to optimize, and I think it's worth doing so especially now that we have a doc value index/skipper.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "gsmiller (via GitHub)" <gi...@apache.org>.
gsmiller commented on PR #13693:
URL: https://github.com/apache/lucene/pull/13693#issuecomment-2343989929

   @iverase if you have a chance to take a look at this, I'd appreciate your thoughts (no rush). I'd be particularly interested what you think about moving the scoring logic of `SortedSetDocValuesRangeQuery` out into `SortedSetDocValuesRangeScorer` so it can be shared with `DocValuesRewriteMethod`. I don't love that part since it means making the scoring logic public to share across sub-packages. (Any other feedback also welcome of course!)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on PR #13693:
URL: https://github.com/apache/lucene/pull/13693#issuecomment-2349297359

   I have my doubts in this approach and I am unsure we should expose the scorer that way. On the other hand, I wonder if we can rewrite the query to a range query. I understand that in this case, you need to be able to apply the optimisation to all segments, but that's probably the most common case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13693:
URL: https://github.com/apache/lucene/pull/13693#issuecomment-2342381311

   This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "gsmiller (via GitHub)" <gi...@apache.org>.
gsmiller commented on PR #13693:
URL: https://github.com/apache/lucene/pull/13693#issuecomment-2351893751

   Thanks so much @iverase for the feedback! I wanted to approach this as a query rewrite initially, but I'm not sure it's all the feasible. The issue is that the ordinal mapping is segment-specific (potentially, at lest), and the rewrite would have to inspect all the segment-level doc values to determine if the range is contiguous. That's the issue I ran into at least. 
   
   I also don't love exposing the scoring implementation publicly. It really is an internal implementation detail. Maybe there's a better way to approach this? I'm open to suggestions. I'm also open to dropping this if we think the exposure isn't work the semi-rare likelihood of the optimization. I just don't like leaving optimizations "on the table" when there's an opportunity to improve.
   
   Maybe I'm overlooking an obvious way to take a query rewrite approach? If so, please let me know. Happy to explore other avenues. Thanks again!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "gsmiller (via GitHub)" <gi...@apache.org>.
gsmiller commented on code in PR #13693:
URL: https://github.com/apache/lucene/pull/13693#discussion_r1733631990


##########
lucene/core/src/java/org/apache/lucene/document/SortedSetDocValuesRangeQuery.java:
##########
@@ -161,66 +156,8 @@ public Scorer get(long leadCost) throws IOException {
               return new ConstantScoreScorer(score(), scoreMode, DocIdSetIterator.empty());
             }
 
-            // all terms matched in this segment

Review Comment:
   This looks like a lot of change but it's mostly just a lift-and-shift into the new `SortedSetDocValuesRangeScorer` class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "gsmiller (via GitHub)" <gi...@apache.org>.
gsmiller commented on code in PR #13693:
URL: https://github.com/apache/lucene/pull/13693#discussion_r1733632247


##########
lucene/core/src/java/org/apache/lucene/document/SortedSetDocValuesRangeQuery.java:
##########
@@ -236,69 +173,4 @@ public boolean isCacheable(LeafReaderContext ctx) {
       }
     };
   }
-
-  private DocIdSetIterator getDocIdSetIteratorOrNullForPrimarySort(

Review Comment:
   Same comment as above (this is all just code that mostly moved as-is into `SortedSetDocValuesRangeScorer`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [PR] Use range optimizations for "slow" MultiTermQueries when terms happen to be contiguous [lucene]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13693:
URL: https://github.com/apache/lucene/pull/13693#issuecomment-3408724259

   This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org