You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "sgup432 (via GitHub)" <gi...@apache.org> on 2026/04/24 17:52:56 UTC
[I] Improve clause ordering for multi-field DocValues range conjunctions [lucene]
sgup432 opened a new issue, #15981:
URL: https://github.com/apache/lucene/issues/15981
### Description
After #15954, SortedNumericDocValuesRangeQuery uses SkipBlockRangeIterator as its two-phase approximation. I see that SkipBlockRangeIterator.cost() currently returns NO_MORE_DOCS, which means when multiple DV range queries are combined in a FILTER conjunction, both DenseConjunctionBulkScorer and ConjunctionDISI will sort the clauses in arbitrary order as they all report the same cost.
I wonder if we should have a better way to do this. That is, we choose the most selective field(the one which can eliminate most docs) as the lead iterator, as this will allow us to skip most of the docs. This might help in performance depending on the number of fields in range conjunctions, higher the no. of fields, better ordering will give better performance.
The DocValuesSkipper already has metadata that could help estimate selectivity ie global minValue()/maxValue(), per-block min/max, and docCount(). Some ideas to do this:
- Use skipper.docCount() as cost. Though only differentiate b/w sparse vs dense fields.
- Estimate selectivity from queryRange / fieldRange * docCount. Rough but differentiates narrow vs wide queries
- Count NO blocks by walking the skip tree at scorer creation. This is more accurate, O(num_blocks) but requires extra work ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Improve clause ordering for multi-field DocValues range conjunctions [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on issue #15981:
URL: https://github.com/apache/lucene/issues/15981#issuecomment-4350762426
I like the idea of estimating how many blocks we can exclude, but we'd need to benchmark it carefully to make sure the overhead isn't too much. Maybe we could only walk the highest level entries? Less accurate, but should be fairly constant time and it will naturally readjust as segments get merged.
We could add a static method to DocValuesSkipper that looks something like:
```
/**
* Returns a value between 0 and 1 inclusive indicating the percentage of skipper blocks
* that overlap the given range.
*/
public static float selectivity(LeafReader reader, String field, long min, long max);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Improve clause ordering for multi-field DocValues range conjunctions [lucene]
Posted by "sgup432 (via GitHub)" <gi...@apache.org>.
sgup432 commented on issue #15981:
URL: https://github.com/apache/lucene/issues/15981#issuecomment-4355458913
>Maybe we could only walk the highest level entries?
Yeah we can start with that/
>We could add a static method to DocValuesSkipper that looks something like:
Hmm makes sense. Let me look into it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Improve clause ordering for multi-field DocValues range conjunctions [lucene]
Posted by "sgup432 (via GitHub)" <gi...@apache.org>.
sgup432 commented on issue #15981:
URL: https://github.com/apache/lucene/issues/15981#issuecomment-4315227111
@romseygeek Can you take a look and provide your suggestions if this makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org