You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "iverase (via GitHub)" <gi...@apache.org> on 2026/05/18 06:36:22 UTC
[PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
iverase opened a new pull request, #16077:
URL: https://github.com/apache/lucene/pull/16077
Iterators generated by RoaringDocIdSet can implement efficiently the method #docIdRunEnd.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #16077:
URL: https://github.com/apache/lucene/pull/16077#discussion_r3257513642
##########
lucene/core/src/test/org/apache/lucene/util/TestRoaringDocIdSet.java:
##########
@@ -36,4 +37,75 @@ public void assertEquals(int numBits, BitSet ds1, RoaringDocIdSet ds2) throws IO
super.assertEquals(numBits, ds1, ds2);
assertEquals(ds1.cardinality(), ds2.cardinality());
}
+
+ public void testDocIDRunEndContiguousWithinBlock() throws IOException {
+ final int maxDoc = 50_000;
+ BitSet bs = new BitSet();
+ for (int d = 12_345; d < 12_360; d++) {
Review Comment:
I create those values randomly now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #16077:
URL: https://github.com/apache/lucene/pull/16077#discussion_r3257515470
##########
lucene/core/src/test/org/apache/lucene/util/TestRoaringDocIdSet.java:
##########
@@ -36,4 +37,75 @@ public void assertEquals(int numBits, BitSet ds1, RoaringDocIdSet ds2) throws IO
super.assertEquals(numBits, ds1, ds2);
assertEquals(ds1.cardinality(), ds2.cardinality());
}
+
+ public void testDocIDRunEndContiguousWithinBlock() throws IOException {
+ final int maxDoc = 50_000;
+ BitSet bs = new BitSet();
+ for (int d = 12_345; d < 12_360; d++) {
+ bs.set(d);
+ }
+ RoaringDocIdSet set = copyOf(bs, maxDoc);
+ assertDocIDRunEndMatches(bs, maxDoc, set);
+ }
+
+ public void testDocIDRunEndAcross64KBoundary() throws IOException {
+ final int boundary = 1 << 16;
+ final int maxDoc = boundary + 100;
+ BitSet bs = new BitSet();
+ for (int d = boundary - 3; d < boundary + 7; d++) {
+ bs.set(d);
+ }
+ RoaringDocIdSet set = copyOf(bs, maxDoc);
+ assertDocIDRunEndMatches(bs, maxDoc, set);
+ }
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase merged PR #16077:
URL: https://github.com/apache/lucene/pull/16077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #16077:
URL: https://github.com/apache/lucene/pull/16077#discussion_r3257520676
##########
lucene/core/src/java/org/apache/lucene/util/NotDocIdSet.java:
##########
@@ -77,6 +77,16 @@ public int advance(int target) throws IOException {
}
}
+ @Override
+ public int docIDRunEnd() throws IOException {
+ assert doc != NO_MORE_DOCS;
+ int skipped = nextSkippedDoc;
+ if (doc > skipped) {
+ skipped = inIterator.advance(doc);
Review Comment:
The contract says:
```
The current position of this iterator is not affected by calling {@link #docIDRunEnd()}
```
And this is not affected, we are just looking ahead if needed by te position of the iterator is the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on code in PR #16077:
URL: https://github.com/apache/lucene/pull/16077#discussion_r3257138645
##########
lucene/CHANGES.txt:
##########
@@ -383,6 +383,9 @@ Optimizations
* GITHUB#16072: Added efficient implementations for BitSetIterator#docIDRunEnd. (Ignacio Vera)
+* GITHUB#16077: Added efficient implementations if the method #docIDRunEnd for the iteerators used
Review Comment:
```suggestion
* GITHUB#16077: Added efficient implementations if the method #docIDRunEnd for the iterators used
```
##########
lucene/core/src/test/org/apache/lucene/util/TestRoaringDocIdSet.java:
##########
@@ -36,4 +37,75 @@ public void assertEquals(int numBits, BitSet ds1, RoaringDocIdSet ds2) throws IO
super.assertEquals(numBits, ds1, ds2);
assertEquals(ds1.cardinality(), ds2.cardinality());
}
+
+ public void testDocIDRunEndContiguousWithinBlock() throws IOException {
+ final int maxDoc = 50_000;
+ BitSet bs = new BitSet();
+ for (int d = 12_345; d < 12_360; d++) {
+ bs.set(d);
+ }
+ RoaringDocIdSet set = copyOf(bs, maxDoc);
+ assertDocIDRunEndMatches(bs, maxDoc, set);
+ }
+
+ public void testDocIDRunEndAcross64KBoundary() throws IOException {
+ final int boundary = 1 << 16;
+ final int maxDoc = boundary + 100;
+ BitSet bs = new BitSet();
+ for (int d = boundary - 3; d < boundary + 7; d++) {
+ bs.set(d);
+ }
+ RoaringDocIdSet set = copyOf(bs, maxDoc);
+ assertDocIDRunEndMatches(bs, maxDoc, set);
+ }
+
Review Comment:
Let's add a test where the run end occurs directly on a boundary?
##########
lucene/core/src/test/org/apache/lucene/util/TestRoaringDocIdSet.java:
##########
@@ -36,4 +37,75 @@ public void assertEquals(int numBits, BitSet ds1, RoaringDocIdSet ds2) throws IO
super.assertEquals(numBits, ds1, ds2);
assertEquals(ds1.cardinality(), ds2.cardinality());
}
+
+ public void testDocIDRunEndContiguousWithinBlock() throws IOException {
+ final int maxDoc = 50_000;
+ BitSet bs = new BitSet();
+ for (int d = 12_345; d < 12_360; d++) {
Review Comment:
Maybe add a comment explaining why we're using these numbers?
##########
lucene/core/src/java/org/apache/lucene/util/NotDocIdSet.java:
##########
@@ -77,6 +77,16 @@ public int advance(int target) throws IOException {
}
}
+ @Override
+ public int docIDRunEnd() throws IOException {
+ assert doc != NO_MORE_DOCS;
+ int skipped = nextSkippedDoc;
+ if (doc > skipped) {
+ skipped = inIterator.advance(doc);
Review Comment:
Doesn't this break the contract that calling docIDRunEnd() doesn't change the current state of the iterator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #16077:
URL: https://github.com/apache/lucene/pull/16077#discussion_r3257535008
##########
lucene/core/src/java/org/apache/lucene/util/NotDocIdSet.java:
##########
@@ -77,6 +77,16 @@ public int advance(int target) throws IOException {
}
}
+ @Override
+ public int docIDRunEnd() throws IOException {
+ assert doc != NO_MORE_DOCS;
+ int skipped = nextSkippedDoc;
+ if (doc > skipped) {
+ skipped = inIterator.advance(doc);
Review Comment:
Damn that's not right. Er are advancing the internal iterator, that's bad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Implement #docIdRunEnd in the iterators used by RoaringDocIdSet [lucene]
Posted by "iverase (via GitHub)" <gi...@apache.org>.
iverase commented on code in PR #16077:
URL: https://github.com/apache/lucene/pull/16077#discussion_r3257638717
##########
lucene/core/src/java/org/apache/lucene/util/NotDocIdSet.java:
##########
@@ -77,6 +77,16 @@ public int advance(int target) throws IOException {
}
}
+ @Override
+ public int docIDRunEnd() throws IOException {
+ assert doc != NO_MORE_DOCS;
+ int skipped = nextSkippedDoc;
+ if (doc > skipped) {
+ skipped = inIterator.advance(doc);
Review Comment:
Add some test for NotDocIdSet. Actually that condition was always false so I added an assert instead that the skipped doc is alway lower than the current doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org