You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/09/12 11:55:13 UTC

[PR] Batch match-all and sorted document-ID iteration [pinot]

xiangfu0 opened a new pull request, #19536:
URL: https://github.com/apache/pinot/pull/19536

   Match-all and sorted-range queries currently fill document blocks through one `next()` call per row. Add a compatible batch API with a scalar default and contiguous fast paths, then use it from `DocIdSetOperator` with the existing scratch array and EOF/checkpoint/accounting behavior.
   
   On a warm, 3,000,017-row fixture with two separately executed server responses, three alternating JMH pairs show 1.65–1.80× sorted-range, 1.13× match-all SUM/AVG, and 1.12× GROUP BY speedups. Bitmap, predicate-scan, metadata COUNT and LIMIT 1 controls are retained; the largest individual pair slowdown is 1.508%. Timing includes SQL preparation, server plans/execution, serialization/deserialization and broker reduction, and excludes HTTP/admission/concurrent-server capacity. Allocation is mostly unchanged. This is not a production QPS or universal all-query speedup claim.
   
   Validation: 148 tests across 19 classes; Spotless, checkstyle, license format/check and Apache RAT; standard compilation/package with compiler warnings reviewed; independent source and benchmark review. The benchmark owns and closes its executor; an earlier leaking-harness pilot was excluded and retained separately. All other runtime bytes and the fixture match across arms.
   
   Reproduction and tutorial: `pinot-perf/README.md`, “Document-ID batch query benchmark,” includes build, fixture generation and JMH commands. Full protocol, controls, all raw measurements and source/runtime manifests are in `pinot-perf/benchmark-results/docid-batching-20260912/`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Batch match-all and sorted document-ID iteration [pinot]

Posted by "Jackie-Jiang (via GitHub) via commits" <co...@pinot.apache.org>.
Jackie-Jiang commented on code in PR #19536:
URL: https://github.com/apache/pinot/pull/19536#discussion_r4621566499


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/DocIdSetOperator.java:
##########
@@ -74,15 +73,10 @@ protected DocIdSetBlock getNextBlock() {
 
     QueryThreadContext.sampleUsage();
 
-    int pos = 0;
     int[] docIds = THREAD_LOCAL_DOC_IDS.get();
-    for (int i = 0; i < _maxSizeOfDocIdSet; i++) {
-      _currentDocId = _blockDocIdIterator.next();
-      if (_currentDocId == Constants.EOF) {
-        break;
-      }
-      docIds[pos++] = _currentDocId;
-    }
+    int pos = _blockDocIdIterator.nextBatch(docIds, _maxSizeOfDocIdSet);
+    // A short batch has already consumed EOF in the scalar fallback. Do not call the iterator again.
+    _exhausted = pos < _maxSizeOfDocIdSet;

Review Comment:
   Can we rename pos to numDocIds now that it represents the returned batch count? The comment can also describe the shared contract directly: "A short batch means the iterator is exhausted; do not call it again."



##########
pinot-core/src/main/java/org/apache/pinot/core/common/BlockDocIdIterator.java:
##########
@@ -29,6 +34,30 @@ public interface BlockDocIdIterator extends AutoCloseable {
   /// [org.apache.pinot.segment.spi.Constants#EOF].
   int next();
 
+  /// Writes up to `maxDocs` next matching document ids into the prefix of the caller-owned `output` array and returns
+  /// the number written. Requires `0 < maxDocs <= output.length`. Does not retain the array or modify its remaining
+  /// elements. A full batch does not consume another document to look ahead.
+  ///
+  /// A return value less than `maxDocs` means the iterator is exhausted. There must be no further calls to
+  /// [#next()], [#advance(int)], or this method after a short batch. Otherwise, these methods share the same cursor
+  /// and may be interleaved subject to the target constraint of [#advance(int)]. This method must not be called after
+  /// either scalar method returns [Constants#EOF].
+  ///
+  /// The default implementation preserves scalar iteration for existing implementations.
+  default int nextBatch(int[] output, int maxDocs) {
+    checkArgument(maxDocs > 0 && maxDocs <= output.length,
+        "maxDocs (%s) must be between 1 and the output length (%s)", maxDocs, output.length);

Review Comment:
   The production caller already guarantees valid bounds: DocIdSetOperator validates the batch size and uses a scratch array sized to MAX_DOC_PER_CALL. We can remove the argument checks and their static imports from all three nextBatch implementations, retain the documented precondition, and remove testInvalidCapacityDoesNotConsumeOrWrite.



##########
pinot-core/src/test/java/org/apache/pinot/core/operator/dociditerators/DocIdIteratorBatchTest.java:
##########
@@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.dociditerators;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+import org.apache.pinot.spi.utils.Pairs.IntPair;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises the shared cursor, caller-owned output and terminal-batch contract of document-id iterators.
+/// Each test uses its own mutable iterator and may run independently.
+public class DocIdIteratorBatchTest {
+  private static final int UNWRITTEN = -123456;
+
+  @DataProvider
+  public Object[][] batchBoundaries() {
+    int maxDocs = DocIdSetPlanNode.MAX_DOC_PER_CALL;
+    return new Object[][]{
+        {1, 0}, {1, 1}, {1, 2}, {2, 1}, {2, 2}, {2, 3},
+        {maxDocs, maxDocs - 1}, {maxDocs, maxDocs}, {maxDocs, maxDocs + 1}
+    };
+  }
+
+  @Test(dataProvider = "batchBoundaries")
+  public void testBatchBoundaries(int capacity, int numDocs) {
+    int[] expected = IntStream.range(0, numDocs).toArray();
+    assertBatches(new MatchAllDocIdIterator(numDocs), capacity, expected);
+    assertBatches(new ScalarIterator(expected), capacity, expected);
+    if (numDocs > 0) {
+      assertBatches(new SortedDocIdIterator(List.of(new IntPair(0, numDocs - 1))), capacity, expected);
+    } else {
+      assertBatches(EmptyDocIdIterator.getInstance(), capacity, expected);
+    }
+  }
+
+  @DataProvider
+  public Object[][] sortedRanges() {
+    return new Object[][]{
+        {List.of(new IntPair(7, 7)), new int[]{7}},
+        {List.of(new IntPair(1, 2), new IntPair(3, 3), new IntPair(4, 6)), new int[]{1, 2, 3, 4, 5, 6}},
+        {List.of(new IntPair(1, 3), new IntPair(8, 11), new IntPair(18, 20)),
+            new int[]{1, 2, 3, 8, 9, 10, 11, 18, 19, 20}},
+        {List.of(new IntPair(Integer.MAX_VALUE - 8, Integer.MAX_VALUE - 7),
+            new IntPair(Integer.MAX_VALUE - 4, Integer.MAX_VALUE - 1)),
+            new int[]{Integer.MAX_VALUE - 8, Integer.MAX_VALUE - 7, Integer.MAX_VALUE - 4,
+                Integer.MAX_VALUE - 3, Integer.MAX_VALUE - 2, Integer.MAX_VALUE - 1}}
+    };
+  }
+
+  @Test(dataProvider = "sortedRanges")
+  public void testSortedRangeTransitions(List<IntPair> ranges, int[] expected) {
+    for (int capacity : new int[]{1, 2, 3, 4, 256}) {
+      assertBatches(new SortedDocIdIterator(ranges), capacity, expected);
+    }
+  }
+
+  @Test
+  public void testSortedScalarAdvanceAndBatchShareCursor() {
+    BlockDocIdIterator iterator = new SortedDocIdIterator(
+        List.of(new IntPair(1, 3), new IntPair(8, 11), new IntPair(18, 20)));
+    assertEquals(iterator.next(), 1);
+    assertBatch(iterator, 2, 2, 3);
+    assertEquals(iterator.advance(5), 8);
+    assertBatch(iterator, 2, 9, 10);
+    assertEquals(iterator.next(), 11);
+    assertBatch(iterator, 1, 18);
+    assertEquals(iterator.advance(20), 20);
+    assertBatch(iterator, 2);
+  }
+
+  @Test
+  public void testMatchAllScalarAdvanceAndBatchShareCursor() {
+    BlockDocIdIterator iterator = new MatchAllDocIdIterator(13);
+    assertEquals(iterator.next(), 0);
+    assertBatch(iterator, 3, 1, 2, 3);
+    assertEquals(iterator.advance(7), 7);
+    assertBatch(iterator, 2, 8, 9);
+    assertEquals(iterator.next(), 10);
+    assertBatch(iterator, 3, 11, 12);
+  }
+
+  @Test
+  public void testMatchAllNearMaximumDocumentId() {
+    BlockDocIdIterator iterator = new MatchAllDocIdIterator(Integer.MAX_VALUE);
+    assertEquals(iterator.advance(Integer.MAX_VALUE - 4), Integer.MAX_VALUE - 4);
+    assertBatch(iterator, 4, Integer.MAX_VALUE - 3, Integer.MAX_VALUE - 2, Integer.MAX_VALUE - 1);
+  }
+
+  @Test
+  public void testDefaultDoesNotLookAheadOrRetainOutput() {
+    ScalarIterator iterator = new ScalarIterator(new int[]{1, 2, 3, 8, 9});
+    int[] output = new int[2];
+    assertEquals(iterator.nextBatch(output, 2), 2);
+    assertEquals(output, new int[]{1, 2});
+    assertEquals(iterator._numNextCalls, 2);
+    // Changing caller-owned storage cannot change the iterator's next result.
+    Arrays.fill(output, UNWRITTEN);
+    assertEquals(iterator.next(), 3);
+    assertEquals(iterator.advance(8), 8);
+    assertBatch(iterator, 2, 9);
+    assertEquals(iterator._numNextCalls, 6);
+  }
+
+  @Test
+  public void testInvalidCapacityDoesNotConsumeOrWrite() {
+    for (BlockDocIdIterator iterator : List.of(new MatchAllDocIdIterator(3),
+        new SortedDocIdIterator(List.of(new IntPair(0, 2))), new ScalarIterator(new int[]{0, 1, 2}))) {
+      int[] output = {UNWRITTEN, UNWRITTEN};
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(output, 0));
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(output, -1));
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(output, 3));
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(new int[0], 1));
+      assertEquals(output, new int[]{UNWRITTEN, UNWRITTEN});
+      assertEquals(iterator.next(), 0);
+    }
+  }
+
+  @DataProvider
+  public Object[][] fallbackIterators() {
+    Supplier<BlockDocIdIterator> bitmap = () -> bitmap(1, 2, 3, 8, 9, 10, 11, 18, 19, 20);
+    Supplier<BlockDocIdIterator> and = () -> new AndDocIdIterator(new BlockDocIdIterator[]{
+        bitmap(1, 2, 3, 8, 9, 10, 18, 19, 20), bitmap(0, 2, 3, 4, 9, 10, 11, 19, 20)});
+    Supplier<BlockDocIdIterator> or = () -> new OrDocIdIterator(new BlockDocIdIterator[]{
+        bitmap(1, 3, 8, 9, 18, 20), bitmap(2, 3, 10, 11, 19, 20)});
+    Supplier<BlockDocIdIterator> not = () -> new NotDocIdIterator(bitmap(0, 3, 7, 9), 12);
+    Supplier<BlockDocIdIterator> notOverOr = () -> new NotDocIdIterator(new OrDocIdIterator(
+        new BlockDocIdIterator[]{bitmap(0, 3, 7, 9), bitmap(1, 3, 8, 10)}), 12);
+    Supplier<BlockDocIdIterator> fullExclusion = () -> new NotDocIdIterator(bitmap(0, 1, 2, 3), 4);
+    return new Object[][]{
+        {bitmap, new int[]{1, 2, 3, 8, 9, 10, 11, 18, 19, 20}},
+        {and, new int[]{2, 3, 9, 10, 19, 20}},
+        {or, new int[]{1, 2, 3, 8, 9, 10, 11, 18, 19, 20}},
+        {not, new int[]{1, 2, 4, 5, 6, 8, 10, 11}},
+        {notOverOr, new int[]{2, 4, 5, 6, 11}},
+        {fullExclusion, new int[0]}
+    };
+  }
+
+  @Test(dataProvider = "fallbackIterators")
+  public void testDefaultFallbacks(Supplier<BlockDocIdIterator> factory, int[] expected) {
+    for (int capacity : new int[]{1, 2, 3, 256}) {
+      assertBatches(factory.get(), capacity, expected);
+    }
+    BlockDocIdIterator scalar = factory.get();
+    for (int docId : expected) {
+      assertEquals(scalar.next(), docId);
+    }
+    assertEquals(scalar.next(), Constants.EOF);
+
+    // Compare scalar and batched progress while alternately skipping and requesting documents.
+    if (expected.length > 5) {
+      BlockDocIdIterator mixed = factory.get();
+      scalar = factory.get();
+      assertEquals(mixed.next(), scalar.next());
+      assertEquals(mixed.advance(expected[2]), scalar.advance(expected[2]));
+      assertBatch(mixed, 2, scalar.next(), scalar.next());
+      assertEquals(mixed.advance(expected[expected.length - 1]), scalar.advance(expected[expected.length - 1]));
+      assertBatch(mixed, 2);
+      assertEquals(scalar.next(), Constants.EOF);
+    }
+  }
+
+  private static BitmapDocIdIterator bitmap(int... docIds) {
+    MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
+    bitmap.add(docIds);
+    return new BitmapDocIdIterator(bitmap, 32);
+  }
+
+  private static void assertBatches(BlockDocIdIterator iterator, int capacity, int[] expected) {
+    int offset = 0;
+    while (true) {
+      int size = Math.min(capacity, expected.length - offset);
+      assertBatch(iterator, capacity, Arrays.copyOfRange(expected, offset, offset + size));
+      offset += size;
+      if (size < capacity) {
+        break;
+      }
+    }
+    assertEquals(offset, expected.length);
+  }
+
+  private static void assertBatch(BlockDocIdIterator iterator, int capacity, int... expected) {
+    int[] output = new int[capacity + 2];
+    Arrays.fill(output, UNWRITTEN);
+    assertEquals(iterator.nextBatch(output, capacity), expected.length);

Review Comment:
   Optional test cleanup: rename capacity to maxDocs in these helpers and callers, since the actual output array is deliberately larger than the requested batch size. We can also remove assertEquals(offset, expected.length) at the end of assertBatches; offset is computed entirely from the expected length and batch size, so that assertion cannot detect an iterator defect.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Batch match-all and sorted document-ID iteration [pinot]

Posted by "xiangfu0 (via GitHub) via commits" <co...@pinot.apache.org>.
xiangfu0 merged PR #19536:
URL: https://github.com/apache/pinot/pull/19536


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Batch match-all and sorted document-ID iteration [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19536:
URL: https://github.com/apache/pinot/pull/19536#issuecomment-5645905514

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19536?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 100.00%. Comparing base ([`5771d6a`](https://app.codecov.io/gh/apache/pinot/commit/5771d6acea60cd72738116835111cf7c49965373?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6208a84`](https://app.codecov.io/gh/apache/pinot/commit/6208a84fbb239808bb65000004eb2386e4c766f4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff               @@
   ##             master    #19536       +/-   ##
   ==============================================
   + Coverage     67.72%   100.00%   +32.27%     
   + Complexity     1450         6     -1444     
   ==============================================
     Files          3490         3     -3487     
     Lines        225032         6   -225026     
     Branches      35527         0    -35527     
   ==============================================
   - Hits         152393         6   -152387     
   + Misses        60612         0    -60612     
   + Partials      12027         0    -12027     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+32.27%)` | :arrow_up: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+32.27%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19536/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19536?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Batch match-all and sorted document-ID iteration [pinot]

Posted by "xiangfu0 (via GitHub) via commits" <co...@pinot.apache.org>.
xiangfu0 commented on code in PR #19536:
URL: https://github.com/apache/pinot/pull/19536#discussion_r4622238757


##########
pinot-core/src/main/java/org/apache/pinot/core/common/BlockDocIdIterator.java:
##########
@@ -29,6 +34,30 @@ public interface BlockDocIdIterator extends AutoCloseable {
   /// [org.apache.pinot.segment.spi.Constants#EOF].
   int next();
 
+  /// Writes up to `maxDocs` next matching document ids into the prefix of the caller-owned `output` array and returns
+  /// the number written. Requires `0 < maxDocs <= output.length`. Does not retain the array or modify its remaining
+  /// elements. A full batch does not consume another document to look ahead.
+  ///
+  /// A return value less than `maxDocs` means the iterator is exhausted. There must be no further calls to
+  /// [#next()], [#advance(int)], or this method after a short batch. Otherwise, these methods share the same cursor
+  /// and may be interleaved subject to the target constraint of [#advance(int)]. This method must not be called after
+  /// either scalar method returns [Constants#EOF].
+  ///
+  /// The default implementation preserves scalar iteration for existing implementations.
+  default int nextBatch(int[] output, int maxDocs) {
+    checkArgument(maxDocs > 0 && maxDocs <= output.length,
+        "maxDocs (%s) must be between 1 and the output length (%s)", maxDocs, output.length);

Review Comment:
   Done in f7ee6ea7e1. I removed the argument checks and their static imports from all three nextBatch implementations, kept the documented precondition on the interface, and dropped testInvalidCapacityDoesNotConsumeOrWrite.



##########
pinot-core/src/main/java/org/apache/pinot/core/operator/DocIdSetOperator.java:
##########
@@ -74,15 +73,10 @@ protected DocIdSetBlock getNextBlock() {
 
     QueryThreadContext.sampleUsage();
 
-    int pos = 0;
     int[] docIds = THREAD_LOCAL_DOC_IDS.get();
-    for (int i = 0; i < _maxSizeOfDocIdSet; i++) {
-      _currentDocId = _blockDocIdIterator.next();
-      if (_currentDocId == Constants.EOF) {
-        break;
-      }
-      docIds[pos++] = _currentDocId;
-    }
+    int pos = _blockDocIdIterator.nextBatch(docIds, _maxSizeOfDocIdSet);
+    // A short batch has already consumed EOF in the scalar fallback. Do not call the iterator again.
+    _exhausted = pos < _maxSizeOfDocIdSet;

Review Comment:
   Done in f7ee6ea7e1. It's numDocIds now, and the comment reads "A short batch means the iterator is exhausted; do not call it again."



##########
pinot-core/src/test/java/org/apache/pinot/core/operator/dociditerators/DocIdIteratorBatchTest.java:
##########
@@ -0,0 +1,251 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.dociditerators;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.function.Supplier;
+import java.util.stream.IntStream;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+import org.apache.pinot.spi.utils.Pairs.IntPair;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises the shared cursor, caller-owned output and terminal-batch contract of document-id iterators.
+/// Each test uses its own mutable iterator and may run independently.
+public class DocIdIteratorBatchTest {
+  private static final int UNWRITTEN = -123456;
+
+  @DataProvider
+  public Object[][] batchBoundaries() {
+    int maxDocs = DocIdSetPlanNode.MAX_DOC_PER_CALL;
+    return new Object[][]{
+        {1, 0}, {1, 1}, {1, 2}, {2, 1}, {2, 2}, {2, 3},
+        {maxDocs, maxDocs - 1}, {maxDocs, maxDocs}, {maxDocs, maxDocs + 1}
+    };
+  }
+
+  @Test(dataProvider = "batchBoundaries")
+  public void testBatchBoundaries(int capacity, int numDocs) {
+    int[] expected = IntStream.range(0, numDocs).toArray();
+    assertBatches(new MatchAllDocIdIterator(numDocs), capacity, expected);
+    assertBatches(new ScalarIterator(expected), capacity, expected);
+    if (numDocs > 0) {
+      assertBatches(new SortedDocIdIterator(List.of(new IntPair(0, numDocs - 1))), capacity, expected);
+    } else {
+      assertBatches(EmptyDocIdIterator.getInstance(), capacity, expected);
+    }
+  }
+
+  @DataProvider
+  public Object[][] sortedRanges() {
+    return new Object[][]{
+        {List.of(new IntPair(7, 7)), new int[]{7}},
+        {List.of(new IntPair(1, 2), new IntPair(3, 3), new IntPair(4, 6)), new int[]{1, 2, 3, 4, 5, 6}},
+        {List.of(new IntPair(1, 3), new IntPair(8, 11), new IntPair(18, 20)),
+            new int[]{1, 2, 3, 8, 9, 10, 11, 18, 19, 20}},
+        {List.of(new IntPair(Integer.MAX_VALUE - 8, Integer.MAX_VALUE - 7),
+            new IntPair(Integer.MAX_VALUE - 4, Integer.MAX_VALUE - 1)),
+            new int[]{Integer.MAX_VALUE - 8, Integer.MAX_VALUE - 7, Integer.MAX_VALUE - 4,
+                Integer.MAX_VALUE - 3, Integer.MAX_VALUE - 2, Integer.MAX_VALUE - 1}}
+    };
+  }
+
+  @Test(dataProvider = "sortedRanges")
+  public void testSortedRangeTransitions(List<IntPair> ranges, int[] expected) {
+    for (int capacity : new int[]{1, 2, 3, 4, 256}) {
+      assertBatches(new SortedDocIdIterator(ranges), capacity, expected);
+    }
+  }
+
+  @Test
+  public void testSortedScalarAdvanceAndBatchShareCursor() {
+    BlockDocIdIterator iterator = new SortedDocIdIterator(
+        List.of(new IntPair(1, 3), new IntPair(8, 11), new IntPair(18, 20)));
+    assertEquals(iterator.next(), 1);
+    assertBatch(iterator, 2, 2, 3);
+    assertEquals(iterator.advance(5), 8);
+    assertBatch(iterator, 2, 9, 10);
+    assertEquals(iterator.next(), 11);
+    assertBatch(iterator, 1, 18);
+    assertEquals(iterator.advance(20), 20);
+    assertBatch(iterator, 2);
+  }
+
+  @Test
+  public void testMatchAllScalarAdvanceAndBatchShareCursor() {
+    BlockDocIdIterator iterator = new MatchAllDocIdIterator(13);
+    assertEquals(iterator.next(), 0);
+    assertBatch(iterator, 3, 1, 2, 3);
+    assertEquals(iterator.advance(7), 7);
+    assertBatch(iterator, 2, 8, 9);
+    assertEquals(iterator.next(), 10);
+    assertBatch(iterator, 3, 11, 12);
+  }
+
+  @Test
+  public void testMatchAllNearMaximumDocumentId() {
+    BlockDocIdIterator iterator = new MatchAllDocIdIterator(Integer.MAX_VALUE);
+    assertEquals(iterator.advance(Integer.MAX_VALUE - 4), Integer.MAX_VALUE - 4);
+    assertBatch(iterator, 4, Integer.MAX_VALUE - 3, Integer.MAX_VALUE - 2, Integer.MAX_VALUE - 1);
+  }
+
+  @Test
+  public void testDefaultDoesNotLookAheadOrRetainOutput() {
+    ScalarIterator iterator = new ScalarIterator(new int[]{1, 2, 3, 8, 9});
+    int[] output = new int[2];
+    assertEquals(iterator.nextBatch(output, 2), 2);
+    assertEquals(output, new int[]{1, 2});
+    assertEquals(iterator._numNextCalls, 2);
+    // Changing caller-owned storage cannot change the iterator's next result.
+    Arrays.fill(output, UNWRITTEN);
+    assertEquals(iterator.next(), 3);
+    assertEquals(iterator.advance(8), 8);
+    assertBatch(iterator, 2, 9);
+    assertEquals(iterator._numNextCalls, 6);
+  }
+
+  @Test
+  public void testInvalidCapacityDoesNotConsumeOrWrite() {
+    for (BlockDocIdIterator iterator : List.of(new MatchAllDocIdIterator(3),
+        new SortedDocIdIterator(List.of(new IntPair(0, 2))), new ScalarIterator(new int[]{0, 1, 2}))) {
+      int[] output = {UNWRITTEN, UNWRITTEN};
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(output, 0));
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(output, -1));
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(output, 3));
+      expectThrows(IllegalArgumentException.class, () -> iterator.nextBatch(new int[0], 1));
+      assertEquals(output, new int[]{UNWRITTEN, UNWRITTEN});
+      assertEquals(iterator.next(), 0);
+    }
+  }
+
+  @DataProvider
+  public Object[][] fallbackIterators() {
+    Supplier<BlockDocIdIterator> bitmap = () -> bitmap(1, 2, 3, 8, 9, 10, 11, 18, 19, 20);
+    Supplier<BlockDocIdIterator> and = () -> new AndDocIdIterator(new BlockDocIdIterator[]{
+        bitmap(1, 2, 3, 8, 9, 10, 18, 19, 20), bitmap(0, 2, 3, 4, 9, 10, 11, 19, 20)});
+    Supplier<BlockDocIdIterator> or = () -> new OrDocIdIterator(new BlockDocIdIterator[]{
+        bitmap(1, 3, 8, 9, 18, 20), bitmap(2, 3, 10, 11, 19, 20)});
+    Supplier<BlockDocIdIterator> not = () -> new NotDocIdIterator(bitmap(0, 3, 7, 9), 12);
+    Supplier<BlockDocIdIterator> notOverOr = () -> new NotDocIdIterator(new OrDocIdIterator(
+        new BlockDocIdIterator[]{bitmap(0, 3, 7, 9), bitmap(1, 3, 8, 10)}), 12);
+    Supplier<BlockDocIdIterator> fullExclusion = () -> new NotDocIdIterator(bitmap(0, 1, 2, 3), 4);
+    return new Object[][]{
+        {bitmap, new int[]{1, 2, 3, 8, 9, 10, 11, 18, 19, 20}},
+        {and, new int[]{2, 3, 9, 10, 19, 20}},
+        {or, new int[]{1, 2, 3, 8, 9, 10, 11, 18, 19, 20}},
+        {not, new int[]{1, 2, 4, 5, 6, 8, 10, 11}},
+        {notOverOr, new int[]{2, 4, 5, 6, 11}},
+        {fullExclusion, new int[0]}
+    };
+  }
+
+  @Test(dataProvider = "fallbackIterators")
+  public void testDefaultFallbacks(Supplier<BlockDocIdIterator> factory, int[] expected) {
+    for (int capacity : new int[]{1, 2, 3, 256}) {
+      assertBatches(factory.get(), capacity, expected);
+    }
+    BlockDocIdIterator scalar = factory.get();
+    for (int docId : expected) {
+      assertEquals(scalar.next(), docId);
+    }
+    assertEquals(scalar.next(), Constants.EOF);
+
+    // Compare scalar and batched progress while alternately skipping and requesting documents.
+    if (expected.length > 5) {
+      BlockDocIdIterator mixed = factory.get();
+      scalar = factory.get();
+      assertEquals(mixed.next(), scalar.next());
+      assertEquals(mixed.advance(expected[2]), scalar.advance(expected[2]));
+      assertBatch(mixed, 2, scalar.next(), scalar.next());
+      assertEquals(mixed.advance(expected[expected.length - 1]), scalar.advance(expected[expected.length - 1]));
+      assertBatch(mixed, 2);
+      assertEquals(scalar.next(), Constants.EOF);
+    }
+  }
+
+  private static BitmapDocIdIterator bitmap(int... docIds) {
+    MutableRoaringBitmap bitmap = new MutableRoaringBitmap();
+    bitmap.add(docIds);
+    return new BitmapDocIdIterator(bitmap, 32);
+  }
+
+  private static void assertBatches(BlockDocIdIterator iterator, int capacity, int[] expected) {
+    int offset = 0;
+    while (true) {
+      int size = Math.min(capacity, expected.length - offset);
+      assertBatch(iterator, capacity, Arrays.copyOfRange(expected, offset, offset + size));
+      offset += size;
+      if (size < capacity) {
+        break;
+      }
+    }
+    assertEquals(offset, expected.length);
+  }
+
+  private static void assertBatch(BlockDocIdIterator iterator, int capacity, int... expected) {
+    int[] output = new int[capacity + 2];
+    Arrays.fill(output, UNWRITTEN);
+    assertEquals(iterator.nextBatch(output, capacity), expected.length);

Review Comment:
   Done in f7ee6ea7e1. The helpers and their callers use maxDocs now, and I renamed the data provider's local to maxDocPerCall so the two don't read as the same thing. I also removed the offset assertion, since the per-batch size check in assertBatch already catches a wrong count.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org