You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/10/08 19:06:12 UTC

[PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

xiangfu0 opened a new pull request, #19799:
URL: https://github.com/apache/pinot/pull/19799

   Repeated short StarTree leaf ranges repeatedly resize buffer array containers. Build ranges in a heap bitmap and convert once for downstream filters, keeping aggregate-only matches on the existing buffer path.
   
   Reproduce: query a StarTree with thousands of one- or two-document leaf ranges in the same 65,536-ID bitmap container while grouping by a dimension below those leaves.
   
   Validation: 18 focused tests and scoped style/license checks pass on JDK 25. Local JMH on a generated 352,832-row mmap segment (three forks/query, full server plan plus serialization) reduces allocation by 35.65% for mixed group-by, 53.92% for residual group-by and 96.04% for range-count, with aggregate-only allocation effectively unchanged and complete row/schema/decoded-wire parity; dense/wider ranges still pay conversion overhead, and general latency gains remain unproven.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19799:
URL: https://github.com/apache/pinot/pull/19799#discussion_r4224836207


##########
pinot-core/src/main/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperator.java:
##########
@@ -233,15 +235,23 @@ private StarTreeResult traverseStarTree() {
 
       // If all predicate columns and group-by columns are matched, we can use aggregated document
       if (remainingPredicateColumns.isEmpty() && remainingGroupByColumns.isEmpty()) {
-        matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        if (heapMatchingDocIds == null) {
+          matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        } else {
+          heapMatchingDocIds.add(starTreeNode.getAggregatedDocId());
+        }
         continue;
       }
 
       // For leaf node, because we haven't exhausted all predicate columns and group-by columns, we cannot use
       // the aggregated document. Add the range of documents for this node to the bitmap, and keep track of the
       // remaining predicate columns for this node
       if (starTreeNode.isLeaf()) {
-        matchingDocIds.add((long) starTreeNode.getStartDocId(), starTreeNode.getEndDocId());
+        if (heapMatchingDocIds == null) {
+          // Range additions grow heap array containers geometrically. Keep aggregate-only matches in the buffer bitmap.
+          heapMatchingDocIds = matchingDocIds.toRoaringBitmap();

Review Comment:
   **[simplification, confirmed]** `matchingDocIds` is provably empty every time this conversion runs, so the comment describes a carry-over that cannot happen. Aggregated doc ids are only added once `remainingPredicateColumns` and `remainingGroupByColumns` are both empty; those sets only shrink (at level transitions), and the aggregated branch `continue`s before the `isLeaf()` check — so once any aggregated doc is added, no leaf-range add can ever follow. The buffer→heap handoff therefore never carries data, no test can cover a non-empty copy, and `toRoaringBitmap()` here is equivalent to `new RoaringBitmap()`.
   
   ```suggestion
             // No doc ids can precede the first leaf range (aggregated docs are only added once all predicate and
             // group-by columns are matched, and those sets only shrink), so start an empty heap bitmap, whose array
             // containers grow geometrically under repeated range additions.
             heapMatchingDocIds = new RoaringBitmap();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19799:
URL: https://github.com/apache/pinot/pull/19799#issuecomment-6067903621

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19799?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 100.00%. Comparing base ([`6f87b71`](https://app.codecov.io/gh/apache/pinot/commit/6f87b7150f40464f455e1a52b2c2a6184f49ccb7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`87d3f26`](https://app.codecov.io/gh/apache/pinot/commit/87d3f26ff583a3ba2d2c4737478a7e908926a266?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff               @@
   ##             master    #19799       +/-   ##
   ==============================================
   + Coverage     68.54%   100.00%   +31.45%     
   + Complexity     1486         6     -1480     
   ==============================================
     Files          3525         3     -3522     
     Lines        229852         6   -229846     
     Branches      36507         0    -36507     
   ==============================================
   - Hits         157558         6   -157552     
   + Misses        59965         0    -59965     
   + Partials      12329         0    -12329     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+31.45%)` | :arrow_up: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+31.45%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19799/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19799?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19799:
URL: https://github.com/apache/pinot/pull/19799#discussion_r4224956647


##########
pinot-core/src/main/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperator.java:
##########
@@ -233,15 +235,23 @@ private StarTreeResult traverseStarTree() {
 
       // If all predicate columns and group-by columns are matched, we can use aggregated document
       if (remainingPredicateColumns.isEmpty() && remainingGroupByColumns.isEmpty()) {
-        matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        if (heapMatchingDocIds == null) {
+          matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        } else {
+          heapMatchingDocIds.add(starTreeNode.getAggregatedDocId());
+        }
         continue;
       }
 
       // For leaf node, because we haven't exhausted all predicate columns and group-by columns, we cannot use
       // the aggregated document. Add the range of documents for this node to the bitmap, and keep track of the
       // remaining predicate columns for this node
       if (starTreeNode.isLeaf()) {
-        matchingDocIds.add((long) starTreeNode.getStartDocId(), starTreeNode.getEndDocId());
+        if (heapMatchingDocIds == null) {
+          // Range additions grow heap array containers geometrically. Keep aggregate-only matches in the buffer bitmap.
+          heapMatchingDocIds = matchingDocIds.toRoaringBitmap();

Review Comment:
   Fixed in c9cedaa462: the first leaf range now uses `new RoaringBitmap()`. The comment documents the monotonic remaining-column invariant: once both sets become empty, every later node takes the aggregate branch before leaf handling. The empty buffer conversion is gone. All 18 selected tests and the 16 public-operator shapes pass.



##########
pinot-core/src/main/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperator.java:
##########
@@ -342,6 +352,9 @@ private StarTreeResult traverseStarTree() {
       }
     }
 
+    if (heapMatchingDocIds != null) {
+      matchingDocIds = heapMatchingDocIds.toMutableRoaringBitmap();

Review Comment:
   Considered both alternatives. Fresh wide ranges create RUN containers in 1.6.23; the updated actual-operator controls measure +264 B/op for one dense range and +416 B/op for three wide ranges/four keys, including conversion. BITMAP containers can still copy 8 KiB/key, so the cost depends on representation. Serialization still copies payload and adds uncached buffer/container views during reads, losing primitive-array fast paths; an N-range gate does not predict representation and needs a populated-buffer handoff. I kept the measured container conversion in this scoped change and retained the overhead caveat. The constructor cleanup is in c9cedaa462; full query row/schema/decoded-wire parity and all 18 selected tests pass.



##########
pinot-core/src/main/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperator.java:
##########
@@ -233,15 +235,23 @@ private StarTreeResult traverseStarTree() {
 
       // If all predicate columns and group-by columns are matched, we can use aggregated document
       if (remainingPredicateColumns.isEmpty() && remainingGroupByColumns.isEmpty()) {
-        matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        if (heapMatchingDocIds == null) {
+          matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        } else {
+          heapMatchingDocIds.add(starTreeNode.getAggregatedDocId());
+        }
         continue;
       }
 
       // For leaf node, because we haven't exhausted all predicate columns and group-by columns, we cannot use
       // the aggregated document. Add the range of documents for this node to the bitmap, and keep track of the
       // remaining predicate columns for this node
       if (starTreeNode.isLeaf()) {
-        matchingDocIds.add((long) starTreeNode.getStartDocId(), starTreeNode.getEndDocId());
+        if (heapMatchingDocIds == null) {
+          // Range additions grow heap array containers geometrically. Keep aggregate-only matches in the buffer bitmap.
+          heapMatchingDocIds = matchingDocIds.toRoaringBitmap();
+        }
+        heapMatchingDocIds.add((long) starTreeNode.getStartDocId(), starTreeNode.getEndDocId());

Review Comment:
   Added the explicit `MappeableArrayContainer.iadd` cause next to the heap initialization in c9cedaa462. I kept this PR scoped to the measured StarTree caller: extracting a public/shared builder and changing sorted-index filtering would add another execution path whose range merging and container-shape tradeoffs need separate measurements. The comment makes the workaround straightforward to revisit after a Roaring upgrade.



##########
pinot-core/src/test/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperatorTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.startree.operator;
+
+import it.unimi.dsi.fastutil.objects.ObjectBooleanPair;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.core.operator.filter.TestUtils;
+import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.startree.CompositePredicateEvaluator;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.startree.StarTree;
+import org.apache.pinot.segment.spi.index.startree.StarTreeNode;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+
+
+/// Tests star-tree document selection through the public filter result. Each invocation owns its fixture.
+public class StarTreeFilterOperatorTest {
+  @DataProvider(name = "nullHandling")
+  public Object[][] nullHandling() {
+    return new Object[][]{{false}, {true}};
+  }
+
+  @Test(dataProvider = "nullHandling")
+  public void testRangesAndAggregatedDocsWithResidualPredicate(boolean nullHandlingEnabled) {
+    // Early leaves retain the dim1 predicate, while deeper dim1 nodes contribute aggregate document ids.
+    // Keep the range order deliberately unsorted, overlap two ranges, and cross the 16-bit container boundary.
+    StarTreeNode firstAggregate = leaf(1, 0, 20, 21, 0);
+    StarTreeNode lastAggregate = leaf(1, 1, 30, 31, 70000);
+    StarTreeNode branch = branch(0, 4, List.of(firstAggregate, lastAggregate));
+    StarTreeNode root = branch(-1, StarTreeNode.ALL, List.of(
+        leaf(0, 0, 65534, 65538, 40), leaf(0, 1, 3, 4, 41), leaf(0, 2, 2, 4, 42),
+        leaf(0, 3, 10, 10, 43), branch));
+    StarTree starTree = mock(StarTree.class);
+    when(starTree.getRoot()).thenReturn(root);
+    when(starTree.getDimensionNames()).thenReturn(List.of("dim0", "dim1"));
+    StarTreeV2Metadata metadata = mock(StarTreeV2Metadata.class);
+    when(metadata.getNumDocs()).thenReturn(70001);
+    StarTreeV2 starTreeV2 = mock(StarTreeV2.class);
+    when(starTreeV2.getStarTree()).thenReturn(starTree);
+    when(starTreeV2.getMetadata()).thenReturn(metadata);
+    QueryContext queryContext = new QueryContext.Builder().build();
+    queryContext.setNullHandlingEnabled(nullHandlingEnabled);
+
+    // A group-by on dim1 takes the same range/singleton paths without applying the residual predicate.
+    StarTreeFilterOperator unfiltered = new StarTreeFilterOperator(queryContext, starTreeV2, Map.of(), Set.of("dim1"));
+    assertEquals(TestUtils.getDocIds(unfiltered.nextBlock().getBlockDocIdSet()),
+        List.of(0, 2, 3, 65534, 65535, 65536, 65537, 70000));
+
+    when(root.getAggregatedDocId()).thenReturn(70000);

Review Comment:
   Fixed in c9cedaa462: the root aggregate id is now stubbed during fixture setup before any operator/assertion. There is no mid-test re-stub, so assertion ordering no longer changes fixture semantics. Both null-handling cases pass.



##########
pinot-core/src/test/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperatorTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.startree.operator;
+
+import it.unimi.dsi.fastutil.objects.ObjectBooleanPair;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.core.operator.filter.TestUtils;
+import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.startree.CompositePredicateEvaluator;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.startree.StarTree;
+import org.apache.pinot.segment.spi.index.startree.StarTreeNode;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+
+
+/// Tests star-tree document selection through the public filter result. Each invocation owns its fixture.
+public class StarTreeFilterOperatorTest {
+  @DataProvider(name = "nullHandling")
+  public Object[][] nullHandling() {
+    return new Object[][]{{false}, {true}};
+  }
+
+  @Test(dataProvider = "nullHandling")
+  public void testRangesAndAggregatedDocsWithResidualPredicate(boolean nullHandlingEnabled) {
+    // Early leaves retain the dim1 predicate, while deeper dim1 nodes contribute aggregate document ids.
+    // Keep the range order deliberately unsorted, overlap two ranges, and cross the 16-bit container boundary.
+    StarTreeNode firstAggregate = leaf(1, 0, 20, 21, 0);
+    StarTreeNode lastAggregate = leaf(1, 1, 30, 31, 70000);
+    StarTreeNode branch = branch(0, 4, List.of(firstAggregate, lastAggregate));
+    StarTreeNode root = branch(-1, StarTreeNode.ALL, List.of(
+        leaf(0, 0, 65534, 65538, 40), leaf(0, 1, 3, 4, 41), leaf(0, 2, 2, 4, 42),
+        leaf(0, 3, 10, 10, 43), branch));
+    StarTree starTree = mock(StarTree.class);
+    when(starTree.getRoot()).thenReturn(root);
+    when(starTree.getDimensionNames()).thenReturn(List.of("dim0", "dim1"));
+    StarTreeV2Metadata metadata = mock(StarTreeV2Metadata.class);
+    when(metadata.getNumDocs()).thenReturn(70001);
+    StarTreeV2 starTreeV2 = mock(StarTreeV2.class);
+    when(starTreeV2.getStarTree()).thenReturn(starTree);
+    when(starTreeV2.getMetadata()).thenReturn(metadata);
+    QueryContext queryContext = new QueryContext.Builder().build();
+    queryContext.setNullHandlingEnabled(nullHandlingEnabled);
+
+    // A group-by on dim1 takes the same range/singleton paths without applying the residual predicate.
+    StarTreeFilterOperator unfiltered = new StarTreeFilterOperator(queryContext, starTreeV2, Map.of(), Set.of("dim1"));
+    assertEquals(TestUtils.getDocIds(unfiltered.nextBlock().getBlockDocIdSet()),
+        List.of(0, 2, 3, 65534, 65535, 65536, 65537, 70000));
+
+    when(root.getAggregatedDocId()).thenReturn(70000);
+    StarTreeFilterOperator aggregateOnly = new StarTreeFilterOperator(queryContext, starTreeV2, Map.of(), null);
+    assertEquals(TestUtils.getDocIds(aggregateOnly.nextBlock().getBlockDocIdSet()), List.of(70000));
+
+    PredicateEvaluator predicateEvaluator = mock(PredicateEvaluator.class);
+    when(predicateEvaluator.getPredicateType()).thenReturn(Predicate.Type.IN);
+    when(predicateEvaluator.getMatchingDictIds()).thenReturn(new int[]{0, 1});
+    CompositePredicateEvaluator compositePredicateEvaluator =
+        new CompositePredicateEvaluator(List.of(ObjectBooleanPair.of(predicateEvaluator, false)));
+    InvertedIndexReader<?> invertedIndex = mock(InvertedIndexReader.class);
+    // Include a matching indexed document outside the selected ranges to prove the residual is conjoined with them.
+    doReturn(MutableRoaringBitmap.bitmapOf(0, 2, 999, 65535, 65536)).when(invertedIndex).getDocIds(0);
+    doReturn(MutableRoaringBitmap.bitmapOf(70000)).when(invertedIndex).getDocIds(1);
+    DataSourceMetadata dataSourceMetadata = mock(DataSourceMetadata.class);
+    when(dataSourceMetadata.isSingleValue()).thenReturn(true);
+    NullValueVectorReader nullValueVector = mock(NullValueVectorReader.class);
+    when(nullValueVector.getNullBitmap()).thenReturn(MutableRoaringBitmap.bitmapOf(65536));
+    DataSource dataSource = mock(DataSource.class);

Review Comment:
   Fixed in c9cedaa462: the fixture now explicitly sets unsorted metadata, column name, absent dictionary/range index, and the inverted index. It also asserts `FilterOperatorUtils.getLeafFilterOperator(...)` selects `InvertedIndexFilterOperator` before exercising the residual conjunction. Both null-handling cases pass and document 999 remains excluded as an out-of-selection indexed match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19799:
URL: https://github.com/apache/pinot/pull/19799#discussion_r4224958169


##########
pinot-core/src/test/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperatorTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.startree.operator;
+
+import it.unimi.dsi.fastutil.objects.ObjectBooleanPair;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.core.operator.filter.TestUtils;
+import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.startree.CompositePredicateEvaluator;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.startree.StarTree;
+import org.apache.pinot.segment.spi.index.startree.StarTreeNode;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+
+
+/// Tests star-tree document selection through the public filter result. Each invocation owns its fixture.
+public class StarTreeFilterOperatorTest {
+  @DataProvider(name = "nullHandling")
+  public Object[][] nullHandling() {
+    return new Object[][]{{false}, {true}};
+  }
+
+  @Test(dataProvider = "nullHandling")
+  public void testRangesAndAggregatedDocsWithResidualPredicate(boolean nullHandlingEnabled) {
+    // Early leaves retain the dim1 predicate, while deeper dim1 nodes contribute aggregate document ids.
+    // Keep the range order deliberately unsorted, overlap two ranges, and cross the 16-bit container boundary.
+    StarTreeNode firstAggregate = leaf(1, 0, 20, 21, 0);

Review Comment:
   Fixed in c9cedaa462: the first aggregate id is now the distinct non-zero value 50, with the expected membership lists and residual bitmap updated. Losing that stub now produces an unexpected 0 instead of silently matching the fixture. Both null-handling cases pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19799:
URL: https://github.com/apache/pinot/pull/19799#discussion_r4224839587


##########
pinot-core/src/main/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperator.java:
##########
@@ -342,6 +352,9 @@ private StarTreeResult traverseStarTree() {
       }
     }
 
+    if (heapMatchingDocIds != null) {
+      matchingDocIds = heapMatchingDocIds.toMutableRoaringBitmap();

Review Comment:
   **[efficiency, plausible]** In RoaringBitmap 1.6.23 this is `new MutableRoaringBitmap(RoaringBitmap)` — a full container-by-container re-materialization (per-container `CharBuffer`/`LongBuffer` allocate + copy) of the entire result on every range-path query. For the shape the PR optimizes (many tiny ranges) this is a clear net win, but for a query matching a few leaf nodes with very wide ranges the old code did 2-3 cheap range adds and zero whole-bitmap copies, while the new code pays one full heap→buffer copy (hundreds of KB for a few-million-doc match) — allocation can regress for that shape, consistent with the “dense/wider ranges still pay conversion overhead” caveat in the description. Two cheaper options worth considering: serialize once and wrap zero-copy (`new ImmutableRoaringBitmap(ByteBuffer.wrap(RoaringBitmapUtils.serialize(heapMatchingDocIds)))` — one contiguous allocation; `ImmutableSegmentImpl` already uses the wrap-serialized-bytes pattern), or only take
  the heap path after N range additions so the few-wide-ranges shape keeps the old single-bitmap behavior.



##########
pinot-core/src/main/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperator.java:
##########
@@ -233,15 +235,23 @@ private StarTreeResult traverseStarTree() {
 
       // If all predicate columns and group-by columns are matched, we can use aggregated document
       if (remainingPredicateColumns.isEmpty() && remainingGroupByColumns.isEmpty()) {
-        matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        if (heapMatchingDocIds == null) {
+          matchingDocIds.add(starTreeNode.getAggregatedDocId());
+        } else {
+          heapMatchingDocIds.add(starTreeNode.getAggregatedDocId());
+        }
         continue;
       }
 
       // For leaf node, because we haven't exhausted all predicate columns and group-by columns, we cannot use
       // the aggregated document. Add the range of documents for this node to the bitmap, and keep track of the
       // remaining predicate columns for this node
       if (starTreeNode.isLeaf()) {
-        matchingDocIds.add((long) starTreeNode.getStartDocId(), starTreeNode.getEndDocId());
+        if (heapMatchingDocIds == null) {
+          // Range additions grow heap array containers geometrically. Keep aggregate-only matches in the buffer bitmap.
+          heapMatchingDocIds = matchingDocIds.toRoaringBitmap();
+        }
+        heapMatchingDocIds.add((long) starTreeNode.getStartDocId(), starTreeNode.getEndDocId());

Review Comment:
   **[altitude, plausible]** The root cause being worked around here is not star-tree-specific: buffer `MappeableArrayContainer.iadd(begin, end)` allocates an exact-size `CharBuffer` on every extending range add (no geometric growth), while heap `ArrayContainer.iadd` grows via `calculateCapacity` — any repeated range-add on a `MutableRoaringBitmap` hits the same quadratic re-allocation. `SortedIndexBasedFilterOperator.getBitmaps()` does the identical per-range `bitmap.add(left, right + 1L)` loop and would benefit equally. If the heap-then-convert trick is worth keeping, consider extracting it into a small shared helper (e.g. next to `RoaringBitmapUtils` in pinot-common) rather than an inline one-off, and naming the upstream cause (`MappeableArrayContainer.iadd`) in a comment so the workaround can be retired if a future RoaringBitmap upgrade fixes the buffer container growth.



##########
pinot-core/src/test/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperatorTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.startree.operator;
+
+import it.unimi.dsi.fastutil.objects.ObjectBooleanPair;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.core.operator.filter.TestUtils;
+import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.startree.CompositePredicateEvaluator;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.startree.StarTree;
+import org.apache.pinot.segment.spi.index.startree.StarTreeNode;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+
+
+/// Tests star-tree document selection through the public filter result. Each invocation owns its fixture.
+public class StarTreeFilterOperatorTest {
+  @DataProvider(name = "nullHandling")
+  public Object[][] nullHandling() {
+    return new Object[][]{{false}, {true}};
+  }
+
+  @Test(dataProvider = "nullHandling")
+  public void testRangesAndAggregatedDocsWithResidualPredicate(boolean nullHandlingEnabled) {
+    // Early leaves retain the dim1 predicate, while deeper dim1 nodes contribute aggregate document ids.
+    // Keep the range order deliberately unsorted, overlap two ranges, and cross the 16-bit container boundary.
+    StarTreeNode firstAggregate = leaf(1, 0, 20, 21, 0);
+    StarTreeNode lastAggregate = leaf(1, 1, 30, 31, 70000);
+    StarTreeNode branch = branch(0, 4, List.of(firstAggregate, lastAggregate));
+    StarTreeNode root = branch(-1, StarTreeNode.ALL, List.of(
+        leaf(0, 0, 65534, 65538, 40), leaf(0, 1, 3, 4, 41), leaf(0, 2, 2, 4, 42),
+        leaf(0, 3, 10, 10, 43), branch));
+    StarTree starTree = mock(StarTree.class);
+    when(starTree.getRoot()).thenReturn(root);
+    when(starTree.getDimensionNames()).thenReturn(List.of("dim0", "dim1"));
+    StarTreeV2Metadata metadata = mock(StarTreeV2Metadata.class);
+    when(metadata.getNumDocs()).thenReturn(70001);
+    StarTreeV2 starTreeV2 = mock(StarTreeV2.class);
+    when(starTreeV2.getStarTree()).thenReturn(starTree);
+    when(starTreeV2.getMetadata()).thenReturn(metadata);
+    QueryContext queryContext = new QueryContext.Builder().build();
+    queryContext.setNullHandlingEnabled(nullHandlingEnabled);
+
+    // A group-by on dim1 takes the same range/singleton paths without applying the residual predicate.
+    StarTreeFilterOperator unfiltered = new StarTreeFilterOperator(queryContext, starTreeV2, Map.of(), Set.of("dim1"));
+    assertEquals(TestUtils.getDocIds(unfiltered.nextBlock().getBlockDocIdSet()),
+        List.of(0, 2, 3, 65534, 65535, 65536, 65537, 70000));
+
+    when(root.getAggregatedDocId()).thenReturn(70000);

Review Comment:
   **[test robustness, plausible]** This mid-test re-stub of the shared `root` mock is only safe because the `unfiltered` assertion above has already eagerly evaluated its traversal via `nextBlock()`, and that traversal never reads root's aggregated doc id. If the assertions are ever reordered, or the first assertion is moved below this line, the fixture semantics silently change across cases instead of failing clearly. Building a fresh root mock (or stubbing the aggregated id up front in `branch()`) per case would remove the ordering dependency.



##########
pinot-core/src/test/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperatorTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.startree.operator;
+
+import it.unimi.dsi.fastutil.objects.ObjectBooleanPair;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.core.operator.filter.TestUtils;
+import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.startree.CompositePredicateEvaluator;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.startree.StarTree;
+import org.apache.pinot.segment.spi.index.startree.StarTreeNode;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+
+
+/// Tests star-tree document selection through the public filter result. Each invocation owns its fixture.
+public class StarTreeFilterOperatorTest {
+  @DataProvider(name = "nullHandling")
+  public Object[][] nullHandling() {
+    return new Object[][]{{false}, {true}};
+  }
+
+  @Test(dataProvider = "nullHandling")
+  public void testRangesAndAggregatedDocsWithResidualPredicate(boolean nullHandlingEnabled) {
+    // Early leaves retain the dim1 predicate, while deeper dim1 nodes contribute aggregate document ids.
+    // Keep the range order deliberately unsorted, overlap two ranges, and cross the 16-bit container boundary.
+    StarTreeNode firstAggregate = leaf(1, 0, 20, 21, 0);
+    StarTreeNode lastAggregate = leaf(1, 1, 30, 31, 70000);
+    StarTreeNode branch = branch(0, 4, List.of(firstAggregate, lastAggregate));
+    StarTreeNode root = branch(-1, StarTreeNode.ALL, List.of(
+        leaf(0, 0, 65534, 65538, 40), leaf(0, 1, 3, 4, 41), leaf(0, 2, 2, 4, 42),
+        leaf(0, 3, 10, 10, 43), branch));
+    StarTree starTree = mock(StarTree.class);
+    when(starTree.getRoot()).thenReturn(root);
+    when(starTree.getDimensionNames()).thenReturn(List.of("dim0", "dim1"));
+    StarTreeV2Metadata metadata = mock(StarTreeV2Metadata.class);
+    when(metadata.getNumDocs()).thenReturn(70001);
+    StarTreeV2 starTreeV2 = mock(StarTreeV2.class);
+    when(starTreeV2.getStarTree()).thenReturn(starTree);
+    when(starTreeV2.getMetadata()).thenReturn(metadata);
+    QueryContext queryContext = new QueryContext.Builder().build();
+    queryContext.setNullHandlingEnabled(nullHandlingEnabled);
+
+    // A group-by on dim1 takes the same range/singleton paths without applying the residual predicate.
+    StarTreeFilterOperator unfiltered = new StarTreeFilterOperator(queryContext, starTreeV2, Map.of(), Set.of("dim1"));
+    assertEquals(TestUtils.getDocIds(unfiltered.nextBlock().getBlockDocIdSet()),
+        List.of(0, 2, 3, 65534, 65535, 65536, 65537, 70000));
+
+    when(root.getAggregatedDocId()).thenReturn(70000);
+    StarTreeFilterOperator aggregateOnly = new StarTreeFilterOperator(queryContext, starTreeV2, Map.of(), null);
+    assertEquals(TestUtils.getDocIds(aggregateOnly.nextBlock().getBlockDocIdSet()), List.of(70000));
+
+    PredicateEvaluator predicateEvaluator = mock(PredicateEvaluator.class);
+    when(predicateEvaluator.getPredicateType()).thenReturn(Predicate.Type.IN);
+    when(predicateEvaluator.getMatchingDictIds()).thenReturn(new int[]{0, 1});
+    CompositePredicateEvaluator compositePredicateEvaluator =
+        new CompositePredicateEvaluator(List.of(ObjectBooleanPair.of(predicateEvaluator, false)));
+    InvertedIndexReader<?> invertedIndex = mock(InvertedIndexReader.class);
+    // Include a matching indexed document outside the selected ranges to prove the residual is conjoined with them.
+    doReturn(MutableRoaringBitmap.bitmapOf(0, 2, 999, 65535, 65536)).when(invertedIndex).getDocIds(0);
+    doReturn(MutableRoaringBitmap.bitmapOf(70000)).when(invertedIndex).getDocIds(1);
+    DataSourceMetadata dataSourceMetadata = mock(DataSourceMetadata.class);
+    when(dataSourceMetadata.isSingleValue()).thenReturn(true);
+    NullValueVectorReader nullValueVector = mock(NullValueVectorReader.class);
+    when(nullValueVector.getNullBitmap()).thenReturn(MutableRoaringBitmap.bitmapOf(65536));
+    DataSource dataSource = mock(DataSource.class);

Review Comment:
   **[test robustness, plausible]** The residual-predicate path routing to an inverted-index-based operator relies entirely on Mockito defaults for unstubbed methods (`isSorted()` = false, `getDictionary()` = null, `getColumnName()` = null — the last tolerated only because the default `QueryContext` has no skip-indexes), and the test never asserts which leaf operator `FilterOperatorUtils` actually selected. If the dispatch precedence changes, the test either dies in `ScanBasedFilterOperator` with a confusing `checkState` failure instead of an assertion diff, or keeps passing through a different operator than the fixture was designed to exercise. Stubbing the routing-relevant methods explicitly (and/or asserting the selected operator type) would make the intent explicit.



##########
pinot-core/src/test/java/org/apache/pinot/core/startree/operator/StarTreeFilterOperatorTest.java:
##########
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.startree.operator;
+
+import it.unimi.dsi.fastutil.objects.ObjectBooleanPair;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.core.operator.filter.TestUtils;
+import org.apache.pinot.core.operator.filter.predicate.PredicateEvaluator;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.apache.pinot.core.startree.CompositePredicateEvaluator;
+import org.apache.pinot.segment.spi.datasource.DataSource;
+import org.apache.pinot.segment.spi.datasource.DataSourceMetadata;
+import org.apache.pinot.segment.spi.index.reader.InvertedIndexReader;
+import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
+import org.apache.pinot.segment.spi.index.startree.StarTree;
+import org.apache.pinot.segment.spi.index.startree.StarTreeNode;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2;
+import org.apache.pinot.segment.spi.index.startree.StarTreeV2Metadata;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertEquals;
+
+
+/// Tests star-tree document selection through the public filter result. Each invocation owns its fixture.
+public class StarTreeFilterOperatorTest {
+  @DataProvider(name = "nullHandling")
+  public Object[][] nullHandling() {
+    return new Object[][]{{false}, {true}};
+  }
+
+  @Test(dataProvider = "nullHandling")
+  public void testRangesAndAggregatedDocsWithResidualPredicate(boolean nullHandlingEnabled) {
+    // Early leaves retain the dim1 predicate, while deeper dim1 nodes contribute aggregate document ids.
+    // Keep the range order deliberately unsorted, overlap two ranges, and cross the 16-bit container boundary.
+    StarTreeNode firstAggregate = leaf(1, 0, 20, 21, 0);

Review Comment:
   **[test clarity, minor]** `firstAggregate`'s aggregated doc id of 0 coincides with Mockito's default int return (so this stub verifies nothing if lost) and doc 0 also appears in the dictId-0 inverted-index bitmap, giving it dual provenance in the filtered assertion. A unique non-zero id (e.g. 50, with the expected lists updated) would make each asserted doc id's provenance unambiguous.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reduce allocation for repeated StarTree leaf ranges [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #19799: Reduce allocation for repeated StarTree leaf ranges
URL: https://github.com/apache/pinot/pull/19799


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org