You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "kkewwei (via GitHub)" <gi...@apache.org> on 2026/08/13 10:35:08 UTC
[PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
kkewwei opened a new pull request, #16510:
URL: https://github.com/apache/lucene/pull/16510
### Description
This PR adds specialized `intoBitSet` implementations for these two cases, so bulk scorers can avoid falling back to per-doc iteration when loading matches into a bit set.
I’m not sure whether additional integration tests are required for these paths. I’d be happy to add them if they would be useful.
<!--
If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide.
https://github.com/apache/lucene/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3914371031
##########
lucene/core/src/test/org/apache/lucene/search/TestIndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -685,6 +690,91 @@ private Query createQuery(String field, long lowerValue, long upperValue) {
field, lowerValue, upperValue, fallbackQuery);
}
+ public void testBoundedIteratorIntoBitSet() throws Exception {
Review Comment:
Agreed, checking whether intoBitSet gets called doesn't provide much value. I've updated the test to check BoundedDocIdSetIterator.intoBitSet directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3914365801
##########
lucene/core/src/test/org/apache/lucene/util/TestNotDocIdSet.java:
##########
@@ -55,6 +78,25 @@ public void testRandomDocIDRunEnd() throws IOException {
}
}
+ public void testIntoBitSetFullBlockComplement() throws IOException {
Review Comment:
Sorry, I didn't notice that a same test already existed, I have deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16510:
URL: https://github.com/apache/lucene/pull/16510#issuecomment-5737913893
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3914365801
##########
lucene/core/src/test/org/apache/lucene/util/TestNotDocIdSet.java:
##########
@@ -55,6 +78,25 @@ public void testRandomDocIDRunEnd() throws IOException {
}
}
+ public void testIntoBitSetFullBlockComplement() throws IOException {
Review Comment:
Sorry, I didn't notice that a same test already existed, I have deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3913129229
##########
lucene/core/src/test/org/apache/lucene/search/TestIndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -685,6 +690,91 @@ private Query createQuery(String field, long lowerValue, long upperValue) {
field, lowerValue, upperValue, fallbackQuery);
}
+ public void testBoundedIteratorIntoBitSet() throws Exception {
Review Comment:
If I'm reading this right, the test is mainly checking two things: that intoBitSet is overridden and that BoundedDocIdSetIterator is what we end up with for sparse ranges. Would it be worth adding unit tests for BoundedDocIdSetIterator directly instead, so we can assert the bits it produces rather than just that the call gets forwarded? It would need to become package private for that, though. WDYT?
##########
lucene/core/src/test/org/apache/lucene/util/TestNotDocIdSet.java:
##########
@@ -55,6 +78,25 @@ public void testRandomDocIDRunEnd() throws IOException {
}
}
+ public void testIntoBitSetFullBlockComplement() throws IOException {
Review Comment:
I'm curious what are the cases that this test covers and testIntoBitSetDenseComplement doesn't?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3914365801
##########
lucene/core/src/test/org/apache/lucene/util/TestNotDocIdSet.java:
##########
@@ -55,6 +78,25 @@ public void testRandomDocIDRunEnd() throws IOException {
}
}
+ public void testIntoBitSetFullBlockComplement() throws IOException {
Review Comment:
Sorry, I didn't notice that a same test already existed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3914349724
##########
lucene/core/src/test/org/apache/lucene/search/TestIndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -685,6 +690,91 @@ private Query createQuery(String field, long lowerValue, long upperValue) {
field, lowerValue, upperValue, fallbackQuery);
}
+ public void testBoundedIteratorIntoBitSet() throws Exception {
Review Comment:
Agreed, checking whether `intoBitSet` gets called doesn't provide much value. I've updated the test to check `BoundedDocIdSetIterator.intoBitSet` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16510:
URL: https://github.com/apache/lucene/pull/16510#issuecomment-5465805923
This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3893844761
##########
lucene/core/src/java/org/apache/lucene/search/IndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -700,5 +701,26 @@ public int advance(int target) throws IOException {
public long cost() {
return Math.min(delegate.cost(), lastDoc - firstDoc);
}
+
+ @Override
+ public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException {
+ assert offset <= doc;
Review Comment:
[nit] Maybe also assert that the bitset is positioned, i.e. doc >= firstDoc?
##########
lucene/core/src/java/org/apache/lucene/util/NotDocIdSet.java:
##########
@@ -84,6 +84,27 @@ public int docIDRunEnd() {
return Math.min(nextSkippedDoc, maxDoc);
}
+ @Override
+ public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException {
+ assert offset <= doc;
+ int boundedUpTo = Math.min(upTo, maxDoc);
+ while (doc < boundedUpTo) {
+ if (doc < nextSkippedDoc) {
+ // [doc, nextSkippedDoc) is a run of matches.
+ int runEnd = Math.min(nextSkippedDoc, boundedUpTo);
+ bitSet.set(doc - offset, runEnd - offset);
+ doc = runEnd;
+ }
+ while (doc == nextSkippedDoc) {
Review Comment:
Given that base class has a requirement `This should behave exactly as if implemented as below` and that `nextDoc() -> advance()` used in the default implementation checks `doc >= maxDoc` on every iteration of its internal loop, should we add this condition to this loop here? Might also make this loop somewhat faster for a corner case where inIterator.nextDoc() keeps returning doc+1 and all of them are greater than maxDoc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3914371031
##########
lucene/core/src/test/org/apache/lucene/search/TestIndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -685,6 +690,91 @@ private Query createQuery(String field, long lowerValue, long upperValue) {
field, lowerValue, upperValue, fallbackQuery);
}
+ public void testBoundedIteratorIntoBitSet() throws Exception {
Review Comment:
Agreed, checking whether intoBitSet gets called doesn't provide much value. I've updated the test to check BoundedDocIdSetIterator.intoBitSet directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3925604442
##########
lucene/core/src/java/org/apache/lucene/search/IndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -700,5 +701,26 @@ public int advance(int target) throws IOException {
public long cost() {
return Math.min(delegate.cost(), lastDoc - firstDoc);
}
+
+ @Override
+ public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException {
+ assert offset <= doc && doc >= firstDoc;
+ int boundedUpTo = Math.min(upTo, lastDoc);
+ if (boundedUpTo > doc) {
+ delegate.intoBitSet(boundedUpTo, bitSet, offset);
+ }
+ int delegateDoc = delegate.docID();
+ if (delegateDoc < lastDoc) {
+ doc = delegateDoc;
+ } else {
+ doc = NO_MORE_DOCS;
+ }
+ }
+
+ @Override
+ public int docIDRunEnd() throws IOException {
Review Comment:
Thank you for the detailed review, LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "epotyom (via GitHub)" <gi...@apache.org>.
epotyom commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3923468648
##########
lucene/core/src/java/org/apache/lucene/search/IndexSortSortedNumericDocValuesRangeQuery.java:
##########
@@ -700,5 +701,26 @@ public int advance(int target) throws IOException {
public long cost() {
return Math.min(delegate.cost(), lastDoc - firstDoc);
}
+
+ @Override
+ public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException {
+ assert offset <= doc && doc >= firstDoc;
+ int boundedUpTo = Math.min(upTo, lastDoc);
+ if (boundedUpTo > doc) {
+ delegate.intoBitSet(boundedUpTo, bitSet, offset);
+ }
+ int delegateDoc = delegate.docID();
+ if (delegateDoc < lastDoc) {
+ doc = delegateDoc;
+ } else {
+ doc = NO_MORE_DOCS;
+ }
+ }
+
+ @Override
+ public int docIDRunEnd() throws IOException {
Review Comment:
Maybe worth adding a small test for this method as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement intoBitSet for NotDocIdSet and IndexSortSortedNumericDocValuesRangeQuery [lucene]
Posted by "kkewwei (via GitHub)" <gi...@apache.org>.
kkewwei commented on code in PR #16510:
URL: https://github.com/apache/lucene/pull/16510#discussion_r3895453409
##########
lucene/core/src/java/org/apache/lucene/util/NotDocIdSet.java:
##########
@@ -84,6 +84,27 @@ public int docIDRunEnd() {
return Math.min(nextSkippedDoc, maxDoc);
}
+ @Override
+ public void intoBitSet(int upTo, FixedBitSet bitSet, int offset) throws IOException {
+ assert offset <= doc;
+ int boundedUpTo = Math.min(upTo, maxDoc);
+ while (doc < boundedUpTo) {
+ if (doc < nextSkippedDoc) {
+ // [doc, nextSkippedDoc) is a run of matches.
+ int runEnd = Math.min(nextSkippedDoc, boundedUpTo);
+ bitSet.set(doc - offset, runEnd - offset);
+ doc = runEnd;
+ }
+ while (doc == nextSkippedDoc) {
Review Comment:
Thanks for the review. Your comment is absolutely correct, I've addressed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org