You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/09/12 22:15:16 UTC

[PR] Reuse partition predicate hashes across broker segments [pinot]

xiangfu0 opened a new pull request, #19539:
URL: https://github.com/apache/pinot/pull/19539

   Broker partition pruning currently revisits the filter tree and rehashes the same literals for every candidate segment. This change prepares visited predicates and reuses literal partition IDs within each pruning call, while checking every segment's current partition set.
   
   Reuse is keyed by partition-function implementation, name, partition count, normalizer and original constructor configuration. Retaining the original configuration also covers plugins that inherit the default null `getFunctionConfig()` implementation. AND/OR/IN keep their evaluation order and short-circuit behavior. Empty/singleton inputs use uncached evaluation; mixed configurations reuse a lookup probe without allocating one key per segment.
   
   ## Benchmark
   
   Independently packaged baseline `5771d6acea60cd72738116835111cf7c49965373` and candidate `8b15fbc6561e39dca7c17d14aa172657a63913db`, with identical benchmark source and bytecode. `BenchmarkPartitionPruning` measures the **complete prune call**, including the returned set's allocation. It cycles through 32 prebuilt requests; parsing, real ZNRecord metadata loading, and independent result-set checks run outside measurement. Homogeneous metadata uses Murmur/128; interleaved metadata alternates Murmur/128 and Murmur/256. Queries combine the measured predicate with two unrelated time filters.
   
   Apple M4 Pro, 24 GiB RAM, Homebrew OpenJDK 25.0.4, JMH 1.37, one benchmark thread, fixed 1 GiB heap, `-XX:ActiveProcessorCount=4`, GC profiler. Serial baseline/PR/PR/baseline order: two independent forks per case per revision, each with five 1-second warmups and five 1-second measurements. For 4,096-segment EQ and IN128, a supplemental ABBA comparison adds two forks per revision with five 2-second warmups and five 1-second measurements, for four forks per revision on those two cases. Every measured iteration is retained; preliminary pilots are excluded.
   
   The following are means across the retained iterations. This shared-workstation benchmark demonstrates local pruning time and allocation; production broker CPU, QPS, and end-to-end latency require workload replay.
   
   | Metadata | Segments | Predicate | Baseline µs/call | PR µs/call | Time reduction | Allocated B/call, baseline → PR |
   |---|---:|---|---:|---:|---:|---:|
   | homogeneous | 1 | EQ | 0.031 | 0.030 | 2.6% | 128 → 128 |
   | homogeneous | 1 | IN_128 | 0.902 | 0.884 | 2.0% | 2503 → 2503 |
   | homogeneous | 1 | UNRELATED | 0.033 | 0.031 | 4.2% | 208 → 208 |
   | homogeneous | 256 | EQ | 6.988 | 2.541 | 63.6% | 8432 → 680 |
   | homogeneous | 256 | IN_128 | 297.725 | 40.575 | 86.4% | 671551 → 12356 |
   | homogeneous | 256 | IN_16 | 62.454 | 8.585 | 86.3% | 125316 → 2528 |
   | homogeneous | 256 | UNRELATED | 7.295 | 5.111 | 29.9% | 12416 → 12800 |
   | homogeneous | 4096 | EQ | 117.634 | 59.751 | 49.2% | 132689 → 2088 |
   | homogeneous | 4096 | IN_128 | 5597.190 | 599.838 | 89.3% | 10743264 → 120997 |
   | homogeneous | 4096 | IN_16 | 933.724 | 178.656 | 80.9% | 2003928 → 24705 |
   | homogeneous | 4096 | UNRELATED | 130.144 | 86.770 | 33.3% | 196801 → 197185 |
   | interleaved | 256 | EQ | 6.430 | 7.225 | -12.4% | 8416 → 1000 |
   | interleaved | 256 | IN_128 | 363.839 | 57.059 | 84.3% | 751245 → 17301 |
   | interleaved | 256 | UNRELATED | 7.467 | 6.799 | 9.0% | 12416 → 13072 |
   
   **Tradeoffs and measurement limits:** Interleaved EQ is 12.4% slower (6.430 → 7.225 µs) while allocating 88.1% fewer bytes. Queries without a partition predicate allocate an extra 384 B/call for homogeneous metadata, or 656 B/call in the interleaved case. Singleton controls are approximately unchanged; their small timing differences should not be read as wins. The baseline's large-segment EQ/IN128 cases vary between JVM forks despite stable iterations within each fork; the per-fork means below expose that variation rather than implying precise confidence in the pooled percentages. The first run began with a host load average of 19.25 after packaging. This task ran no concurrent builds or benchmark JVMs.
   
   <details>
   <summary>Per-fork means (µs per complete pruning call)</summary>
   
   | Metadata | Segments | Predicate | Baseline forks A / B / C / D | PR forks A / B / C / D |
   |---|---:|---|---|---|
   | homogeneous | 1 | EQ | 0.030 / 0.032 | 0.030 / 0.030 |
   | homogeneous | 1 | IN_128 | 0.894 / 0.910 | 0.881 / 0.887 |
   | homogeneous | 1 | UNRELATED | 0.033 / 0.032 | 0.032 / 0.031 |
   | homogeneous | 256 | EQ | 7.267 / 6.709 | 2.523 / 2.558 |
   | homogeneous | 256 | IN_128 | 312.676 / 282.774 | 40.851 / 40.299 |
   | homogeneous | 256 | IN_16 | 62.223 / 62.686 | 8.478 / 8.692 |
   | homogeneous | 256 | UNRELATED | 7.226 / 7.364 | 4.822 / 5.400 |
   | homogeneous | 4096 | EQ | 95.980 / 124.076 / 136.227 / 114.252 | 60.270 / 59.315 / 59.386 / 60.034 |
   | homogeneous | 4096 | IN_128 | 3737.818 / 5998.606 / 6369.155 / 6283.183 | 597.865 / 604.576 / 599.421 / 597.491 |
   | homogeneous | 4096 | IN_16 | 934.902 / 932.546 | 174.327 / 182.985 |
   | homogeneous | 4096 | UNRELATED | 129.133 / 131.155 | 83.924 / 89.615 |
   | interleaved | 256 | EQ | 6.601 / 6.259 | 7.224 / 7.225 |
   | interleaved | 256 | IN_128 | 357.585 / 370.093 | 56.524 / 57.594 |
   | interleaved | 256 | UNRELATED | 7.575 / 7.360 | 6.803 / 6.796 |
   
   </details>
   
   ## Reproduce
   
   Copy `BenchmarkPartitionPruning.java` into a separate worktree at the baseline commit. Package each worktree independently with JDK 25:
   
   ```sh
   ./mvnw -pl pinot-perf -am package -Ppinot-fastdev -DskipTests \
     -Dskip.npm -Dskip.installnodenpm -Denforcer.skip \
     -Dassembly.skipAssembly=true -Dmaven.javadoc.skip=true
   ```
   
   Run each group in each worktree in baseline/PR/PR/baseline order, with distinct output filenames. Verify the packaged broker/perf jars match their module target jars before running.
   
   ```sh
   "$JAVA_HOME/bin/java" -Xmx1g \
     -cp 'pinot-perf/target/pinot-perf-pkg/lib/*' org.openjdk.jmh.Main \
     BenchmarkPartitionPruning -wi 5 -i 5 -w 1s -r 1s -f 1 -t 1 \
     -p _metadata=HOMOGENEOUS -p _numSegments=256,4096 \
     -p _predicate=EQ,IN_16,IN_128,UNRELATED \
     -prof gc -foe true -jvmArgs '-Xms1g -Xmx1g -XX:ActiveProcessorCount=4' \
     -rf json -rff results.json
   ```
   
   For singleton controls, use `_numSegments=1` and `_predicate=EQ,IN_128,UNRELATED`. For interleaved controls, use `_metadata=INTERLEAVED`, `_numSegments=256`, and `_predicate=EQ,IN_128,UNRELATED`. Repeat the additional ABBA comparison for `_metadata=HOMOGENEOUS`, `_numSegments=4096`, `_predicate=EQ,IN_128`, and `-w 2s`.
   
   ## Validation
   
   - 22 tests passed, zero failures/errors/skips: `SinglePartitionColumnSegmentPrunerTest` (11 cases), `SegmentPrunerTest` (8), `SegmentPartitionMetadataManagerTest` (3).
   - Hash-count assertions cover reuse across distinct function objects and interleaved configurations. Tests also cover hidden plugin config, normalizers, raw-byte hashing, lazy errors in both cached/uncached paths, missing metadata, refresh, and concurrent-query isolation.
   - Broker/perf `spotless:apply`, `license:format`, `checkstyle:check`, and `license:check` passed.
   - Warning/deprecation-enabled reactor `test-compile` passed, with no warnings in changed sources.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4190949930


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   Correction to my earlier explanation: the multi-partition example demonstrates conservative segment retention, not incorrect query results. For a single-valued partition column, a row satisfying an AND must have one partition ID in every child's candidate set. Intersecting those sets is safe even when a segment contains several partitions. My claim that the intersection was incorrect was wrong.
   
   The updated PR now eagerly prepares the whole supported predicate once per equivalent partition function, using intersection for AND and union for OR. It then checks each segment against the resulting ID set. If preparation throws on an invalid literal or expression, that function group falls back to the original per-segment evaluator so its short-circuit/error behavior is preserved. The tests now require pruning the impossible conjunction and cover nested intersections, unions, and the error fallback.
   



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   You are right. My earlier multi-partition example demonstrated conservative segment selection, not an incorrect result. A matching row in a single-valued partition column has one ID that must belong to every AND candidate set. I corrected the earlier comment.
   
   Commit 850bad9f22 now eagerly prepares the whole supported predicate once per equivalent function per query, intersecting AND sets and unioning OR sets. The previous assertion requiring the impossible conjunction to be retained has been replaced with an assertion that it is pruned. Tests also cover nested set operations and an invalid literal after an empty intersection.
   
   Error policy: if eager preparation throws a RuntimeException, that function group is marked for the original per-segment evaluator for the rest of the prune call. This preserves the old evaluator's short-circuit/error behavior without paying repeated preparation attempts. Valid predicates use the prepared set and singleton fast path. All 40 focused broker cases pass, along with the downstream Custom fixture.
   
   The bounded JMH comparison is against abbba34a77 on JDK 25: one thread, two forks, 3 × 1-second warmups and 5 × 1-second measurements. The fixture uses independently constructed equal 64-way Modulo functions and one partition per segment. First-hit IN uses partition 0 throughout; other fixtures cycle through partitions. Mean ± JMH-reported error, µs per full prune call:
   
   | Filter | Segments | Before (µs) | Eager (µs) | Change | Interpretation |
   |---|---:|---:|---:|---:|---|
   | EQUALS | 256 | 1.66 ± 0.13 | 1.66 ± 0.41 | -0.2% | Inconclusive |
   | EQUALS | 512 | 3.71 ± 0.11 | 3.23 ± 0.22 | -12.8% | Inconclusive |
   | EQUALS | 1024 | 9.06 ± 1.05 | 9.26 ± 3.58 | +2.2% | Inconclusive |
   | IN(16) | 256 | 2.66 ± 0.07 | 3.50 ± 1.04 | +31.5% | Inconclusive |
   | IN(16) | 512 | 5.77 ± 0.18 | 5.23 ± 0.21 | -9.4% | Likely change |
   | IN(16) | 1024 | 12.21 ± 0.34 | 10.96 ± 0.45 | -10.2% | Likely change |
   | Nested AND/OR | 256 | 3.67 ± 0.09 | 3.06 ± 0.07 | -16.6% | Likely change |
   | Nested AND/OR | 512 | 7.68 ± 0.41 | 5.26 ± 0.28 | -31.4% | Likely change |
   | Nested AND/OR | 1024 | 16.06 ± 0.45 | 11.49 ± 1.15 | -28.5% | Likely change |
   | IN(128), first hit | 256 | 3.96 ± 0.40 | 4.90 ± 0.07 | +23.8% | Likely change |
   | IN(128), first hit | 512 | 8.35 ± 0.30 | 8.99 ± 0.16 | +7.7% | Likely change |
   | IN(128), first hit | 1024 | 19.48 ± 9.05 | 17.85 ± 1.15 | -8.3% | Inconclusive |
   
   The IN(16)/256 repeat was 3.97 ± 0.09 µs, so there is a real small-case cost rather than a uniform speedup. First-hit IN also adds about 0.94/0.65 µs at 256/512. Nested preparation improves about 17/31/28% at 256/512/1024. I kept the eager design and documented the tradeoffs and noisy controls in the PR description. No benchmark source is committed; CI on the updated commits is running.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #19539:
URL: https://github.com/apache/pinot/pull/19539#issuecomment-6031104240

   Docs follow-up: https://github.com/pinot-contrib/pinot-docs/pull/1089 (merged). The broker, table-routing, and query-option references now cover partitionPruningPreparationThreshold, its precedence and default, and the negative-value disable behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4042064025


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;

Review Comment:
   Updated in 0904f9de5a: the query-local cache keeps IDs for the visited literal prefix in order, without deduplicating them into an IntSet. Later compatible segments resume that prefix; incompatible functions neither read nor extend it. This preserves short-circuit behavior and avoids eagerly evaluating remaining IN values. The existing duplicate-partition test now also crosses configured/default functions and checks configured-first ordering. Leaving the optional deduplication proposal open.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {

Review Comment:
   Updated in 0904f9de5a: the prepared-predicate class has returned for large inputs. Child filter decoding and literal conversion remain deferred until visited so skipped AND/OR/IN branches do not start throwing errors that the original evaluator avoided. Existing tests exercise these lazy-error paths below and above the cutoff; 26 broker and 3 routing integration cases pass. Reopening this constructor/laziness discussion because the prior reply described a class removal that no longer applies.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   Updated in 0904f9de5a: removed PartitionFunctionKey and its map entirely. SegmentPartitionInfo and SegmentPartitionUtils remain unchanged. Each prune call keeps only a local first-function reference for compatible default-config ID reuse; mismatched/configured functions compute their own IDs without comparing configuration contents. Leaving the design preference open for review.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   Updated in 0904f9de5a: empty candidates return immediately; fewer than 256 segments use the original evaluator and collection loop. Larger inputs lazily prepare the filter once per prune call. The local comparison shows 85–86% faster large homogeneous IN cases, but a repeated singleton-IN cost of about 25 ns (2.9%) remains and is an accepted tradeoff. The PR description includes methodology, slower controls, and allocation costs. This does reintroduce a separate prepared evaluator, so I am reopening the maintenance/design concern for review rather than treating the earlier class removal as its resolution.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {

Review Comment:
   Updated in 0904f9de5a: isPartitionMatch() is restored to the original recursive evaluator. Inputs below 256 segments use that evaluator and the original collection loop. Larger inputs use a separate query-local prepared predicate. No segment metadata changes.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   Updated in 0904f9de5a: configuration contents are never compared. Nonempty-config functions do not share cached IDs; large inputs still reuse the decoded filter structure. No metadata canonicalization or cross-query state was added. With independently deserialized 100 KB BoundedColumnValue configs across 1,000 segments, the local uncached-baseline comparison measured equality 45.6493→35.1011 us/op (-23.1%), IN 246.8131→239.2255 (-3.1%), and unrelated predicates 31.1160→28.7301 (-7.7%). Methodology and allocation tradeoffs are in the PR description. Leaving the thread open for review.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4190590634


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   I reran a bounded full-prune JMH comparison on the updated code: JDK 25, 1 thread, 2 forks, 3 × 1-second warmups and 5 × 1-second measurements per fork. The fixture has independently constructed, equal 64-way Modulo functions and one partition per segment; times below are µs per complete `prune()` call (mean ± JMH-reported error). “Legacy” disables preparation, “Lazy” is this PR, and “Full eager” is a temporary, uncommitted whole-predicate ID-set variant. The eager fixture's selected segments were checked against legacy before timing.
   
   | Filter | Segments | Legacy | Lazy (PR) | Full eager (temporary) |
   |---|---:|---:|---:|---:|
   | EQUALS | 256 | 3.12 ± 0.12 | 1.70 ± 0.07 | 1.69 ± 0.07 |
   | EQUALS | 4,096 | 62.28 ± 2.42 | 34.73 ± 0.95 | 36.07 ± 4.71 |
   | EQUALS | 16,384 | 299.61 ± 34.65 | 150.09 ± 2.16 | 166.56 ± 40.87 |
   | IN(16) | 256 | 38.13 ± 0.57 | 2.60 ± 0.03 | 2.59 ± 0.48 |
   | IN(16) | 4,096 | 589.78 ± 13.44 | 48.70 ± 1.15 | 48.13 ± 4.66 |
   | IN(16) | 16,384 | 2311.28 ± 408.49 | 257.33 ± 109.77 | 223.36 ± 58.37 |
   | IN(128), mixed | 256 | 67.69 ± 2.51 | 5.59 ± 0.16 | 5.63 ± 2.12 |
   | IN(128), mixed | 4,096 | 1156.41 ± 94.86 | 92.29 ± 11.55 | 79.59 ± 2.69 |
   | IN(128), mixed | 16,384 | 5102.21 ± 693.90 | 432.47 ± 58.20 | 392.26 ± 51.51 |
   | Nested AND/OR | 256 | 48.06 ± 8.61 | 6.39 ± 0.13 | 2.77 ± 0.04 |
   | Nested AND/OR | 4,096 | 728.26 ± 59.33 | 118.25 ± 0.84 | 45.67 ± 0.91 |
   | Nested AND/OR | 16,384 | 2873.04 ± 728.63 | 512.92 ± 9.29 | 202.74 ± 5.74 |
   | IN(128), first-hit | 256 | 5.28 ± 0.04 | 3.90 ± 0.32 | 5.10 ± 0.09 |
   | IN(128), first-hit | 4,096 | 95.96 ± 5.07 | 67.43 ± 4.02 | 66.36 ± 2.15 |
   | IN(128), first-hit | 16,384 | 445.49 ± 13.12 | 335.03 ± 13.89 | 314.49 ± 18.70 |
   
   The larger-segment and nested results support your point: at 4,096 and 16,384 segments, full eager reduces the nested case from 118.25 to 45.67 and from 512.92 to 202.74 µs/op. But on a true first-literal-hit IN(128) case at 256 segments, it regresses from 3.90 to 5.10 µs/op (+31%); the 4,096/16,384 differences overlap reported error. The 16,384-segment IN(16) and some other wide-filter cells are noisy, so I would not use them to claim a precise crossover.
   
   Yes, the lazy path already caches `_singlePartitionId` once per equal function for EQUALS and single-value IN. Fully evaluating rejected literals/branches would also change the existing short-circuit/error behavior (covered by the invalid-literal tests). Given that and the small-case regression, I kept the minimal lazy implementation here. This is a synthetic broker-pruner microbenchmark, not Kraken end-to-end QPS; no benchmark source is committed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #19539:
URL: https://github.com/apache/pinot/pull/19539


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4111440073


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,35 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    List<PreparedPredicate> predicates = new ArrayList<>();
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      int functionHashCode = function.hashCode();
+      PreparedPredicate predicate = null;
+      for (PreparedPredicate candidate : predicates) {

Review Comment:
   I kept preparation lazy because eagerly building a whole-predicate ID set would evaluate literals and branches the current pruner short-circuits. For example, `IN (1, 'invalid-number')` can match before parsing the second value. The query-local results are now indexed by `PartitionFunction` in a `HashMap`, so segment lookup no longer scans a growing list. The existing tests cover lazy evaluation and mixed functions.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java:
##########
@@ -209,6 +213,37 @@ public void init(HelixManager helixManager) {
     _propertyStore = helixManager.getHelixPropertyStore();
   }
 
+  @Override
+  public void onChange(Set<String> changedConfigs, Map<String, String> clusterConfigs) {
+    String key = CommonConstants.Broker.CONFIG_OF_PARTITION_PRUNING_MIN_SEGMENTS;
+    int updatedThreshold = CommonConstants.Broker.DEFAULT_PARTITION_PRUNING_MIN_SEGMENTS;
+    String value = clusterConfigs.get(key);
+    if (value != null) {
+      try {
+        updatedThreshold = Integer.parseInt(value);
+      } catch (NumberFormatException e) {
+        LOGGER.warn("Ignoring invalid partition pruning threshold: {}={}", key, value);
+      }
+    }
+    int previousThreshold = _partitionPruningMinSegments;
+    _partitionPruningMinSegments = updatedThreshold;
+    if (previousThreshold == updatedThreshold) {
+      return;
+    }
+    for (String tableNameWithType : _routingEntryMap.keySet()) {

Review Comment:
   Done. The manager holds a volatile global threshold, and existing pruners read it once at the start of each prune call. Table and query overrides still take precedence, and a cluster update no longer rebuilds routing entries. I added a test that changes the threshold on an already-created pruner.



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -339,6 +339,10 @@ public static class Instance {
   }
 
   public static class Broker {
+    /// Minimum candidate segments for preparing a partition-pruning predicate. A negative value disables preparation.
+    public static final String CONFIG_OF_PARTITION_PRUNING_MIN_SEGMENTS =

Review Comment:
   Done. The new cluster, table, and query settings are now consistently named `partitionPruningPreparationThreshold`; the cluster key is `pinot.broker.partition.pruning.preparation.threshold`. The default remains 32, and a negative value disables preparation only.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -58,11 +58,19 @@ default List<String> getNames() {
   /// @return Number of possible partitions.
   int getNumPartitions();
 
+  /// Returns the configuration used to construct this function, or null if none is exposed.
   @Nullable
   default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether this function and `other` produce identical partition ids for every input.
+  /// Implementations that override [#equals(Object)] must include every setting that affects partition ids.
+  /// Matching hash codes alone are insufficient.
+  default boolean canReusePartitionIds(PartitionFunction other) {

Review Comment:
   Done. I removed `canReusePartitionIds` and keyed prepared predicates directly by `PartitionFunction`; `HashMap` uses `hashCode` and `equals` for lookup. The tests now assert equality directly, and the SPI documents that equality must include all partition-ID-affecting state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4190584149


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java:
##########
@@ -137,6 +139,8 @@ public abstract class BaseBrokerRoutingManager implements RoutingManager, Cluste
   private final Set<String> _excludedServers = new HashSet<>();
   private final ServerRoutingStatsManager _serverRoutingStatsManager;
   private final PinotConfiguration _pinotConfig;
+  private volatile int _partitionPruningPreparationThreshold =

Review Comment:
   Moved `_partitionPruningPreparationThreshold` into the mutable-field section, immediately before `_routableServerInstanceMap`, in 096db71a46.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4042063613


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {

Review Comment:
   Simplified in eb06cf5f47: restored the original recursive isPartitionMatch() evaluator. The only functional addition is a query-local literal-to-partition-ID cache; equality and IN share that lookup. Removed the prepared-predicate tree and all metadata changes.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   Re-scoped in eb06cf5f47 to keep this optimization entirely within one prune call. SegmentPartitionInfo and SegmentPartitionUtils are unchanged from the base. One small record is used only as the local map key to distinguish function class, name, partition count, normalizer, and config. Reopening this design preference for review rather than retaining the broader metadata changes.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   In eb06cf5f47, singleton and multi-segment inputs use the original evaluator with a cache local to prune(). Empty candidates return immediately. There is no QueryPartitionMatcher or prepared-predicate tree.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {

Review Comment:
   The prepared-predicate class and its initialization/lazy-child machinery have been removed in eb06cf5f47. The original recursive evaluator naturally preserves AND/OR short-circuiting, and partition IDs are computed only for visited values. All 24 focused broker tests and 3 partition-routing integration cases pass.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;

Review Comment:
   The narrowed eb06cf5f47 implementation caches each visited literal's partition ID and retains the existing IN loop. It intentionally does not add a deduplicated IntSet or incremental-evaluation state. Duplicate-partition inputs remain correct and each visited literal is computed once per function configuration per prune call. Reopening this optional optimization because the prior IntSet implementation was removed.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   In eb06cf5f47 the change is deliberately limited to a query-local partition-ID cache; metadata canonicalization has been removed. The cache is now accessed only after identifying an equality/IN predicate on the partition column, so unrelated predicates no longer compare configs. Matching partition predicates still use full configuration equality, so the large-config comparison concern remains. Reopening this non-blocking concern; no benchmark claim is made for the smaller implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4190618823


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   I don't follow why you still pick lazy path with this benchmark. Full eager almost beats lazy in all scenarios, except for the cheap ones, where the actual overhead is probably negligible. The saving is huge for nested AND/OR, which is very common



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4201927563


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -58,6 +62,13 @@ default List<String> getNames() {
   /// @return Number of possible partitions.
   int getNumPartitions();
 
+  /// Whether query-local predicate preparation is useful for this function. Functions whose instances cannot be
+  /// compared cheaply across segments can return false to use the original per-segment pruning path.
+  default boolean supportsPartitionIdPreparation() {
+    return true;

Review Comment:
   Done in e36cc19841. The SPI capability now defaults to false; the six built-in functions with value equality explicitly opt in. BoundedColumnValue and legacy identity-only plugins retain the original per-segment path. Unknown subclasses also stay on that path unless they explicitly opt in.
   
   Added a factory-loaded identity-only plugin regression test with 32 segments and 128 IN literals: it performs 32 partition calculations, one first hit per segment, without preparing or retaining per-segment candidate sets. The existing equality/call-count tests still verify reuse for opted-in functions. The new tests fail against 850bad9f22 and pass with this fix.
   
   The companion StarTree update ea9777b2c1 explicitly opts Custom, Iceberg Bucket, and Iceberg Truncate into preparation and keeps unknown subclasses opted out. Validation passed 31 Apache partition-function cases, 42 broker-pruner cases, and 56 StarTree function cases plus the cross-repository Custom pruning test. Module formatting, Checkstyle, and license checks pass. Fresh CI is running.
   



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +237,107 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Computes conservative candidate IDs for the whole predicate. Null represents every partition.
+  /// Partition columns are single-valued, so a row satisfying AND must belong to every child's candidate set.
+  @Nullable
+  private IntSet preparePartitionIds(Expression expression, PartitionFunction partitionFunction) {
+    Function function = expression.getFunctionCall();
+    FilterKind kind = FilterKind.valueOf(function.getOperator());
+    List<Expression> operands = function.getOperands();
+    switch (kind) {
+      case AND: {
+        IntSet ids = null;
+        for (Expression child : operands) {
+          IntSet childIds = preparePartitionIds(child, partitionFunction);
+          if (childIds != null) {
+            if (ids == null) {
+              ids = childIds;
+            } else {
+              // Singleton leaves are immutable; composite and IN sets can be intersected in place.
+              if (!(ids instanceof IntOpenHashSet)) {
+                ids = new IntOpenHashSet(ids);
+              }
+              ids.retainAll(childIds);
+            }
+          }
+        }
+        return ids;
+      }
+      case OR: {
+        IntSet ids = new IntOpenHashSet();
+        boolean matchesAll = false;
+        for (Expression child : operands) {
+          IntSet childIds = preparePartitionIds(child, partitionFunction);
+          if (childIds == null) {
+            matchesAll = true;

Review Comment:
   Done in e36cc19841. OR preparation now returns null immediately when a child is unrestricted; the matchesAll flag is removed. Eager preparation still combines reachable AND/OR branches, but an unrestricted OR child makes later siblings unreachable, just as in the legacy evaluator.
   
   The counting-function regression test uses an unrelated first child followed by 128 IN literals (including an invalid final literal). All segments are retained with zero partition calculations and no error. Against 850bad9f22 it made 128 unnecessary calculations; with the fix it makes zero. All 42 focused broker-pruner cases pass, including the existing short-circuit/error coverage.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4041112691


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   Is preparing a `QueryPartitionMatcher` more expensive than doing `isPartitionMatch()`? We can short circuit zero segment case (if it can reach here). The benefit is that we only need to maintain one logic.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {

Review Comment:
   (minor) No need to change this method, seems more readable before



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   Why do we need to introduce this wrapper? Why not just implement them into `SegmentPartitionInfo`?



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {

Review Comment:
   Do we need this laziness? Seems it is always needed immediately after constructor. Putting it into constructor is easier to read, and everything can be final



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;

Review Comment:
   Should we maintain it as an `IntSet`? For `IN`, multiple values could map to the same partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4186284174


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java:
##########
@@ -137,6 +139,8 @@ public abstract class BaseBrokerRoutingManager implements RoutingManager, Cluste
   private final Set<String> _excludedServers = new HashSet<>();
   private final ServerRoutingStatsManager _serverRoutingStatsManager;
   private final PinotConfiguration _pinotConfig;
+  private volatile int _partitionPruningPreparationThreshold =

Review Comment:
   (minor) Move this to the non-final variables part, probably before `_routableServerInstanceMap`



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -30,6 +30,8 @@
 /// Implementations of this interface are assumed not to be stateful. That is, two invocations of
 /// `PartitionFunction.getPartition(value)` with the same value are expected to produce the same
 /// result. Implementations must also be safe for concurrent invocation by multiple threads.
+/// Functions with equal identity may reuse partition ids during broker pruning. Implementations that override

Review Comment:
   We should add javadoc stating that the function config is not included in the computation of equals and hashCode. We only compare the fields related to partition id computation



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   > I tried the whole-predicate ID-set approach in a temporary full-prune JMH variant (256 segments, one or two functions; 3x1s warmups, 5x1s measurements, two forks on JDK 25). It helps wide filters: homogeneous IN(16) went from 5.70 to 3.10 us/op and OR(4) from 6.43 to 2.73. But it regressed the common EQUALS case from 2.46 to 2.99 (+21%) and early-hit IN(128) from 5.86 to 6.70 (+14%; two functions: 6.49 to 9.56, +47%). Omitting rejected literals also changes the existing short-circuit/error behavior (covered by the invalid-number tests); preserving it needs a fallback path. Given those regressions and the extra path, I kept lazy preparation for this minimal PR. No benchmark source is committed.
   
   Can you try a benchmark with more segments? 256 segments are just too few to show the performance benefit of fully preparing the predicate. Also, do you have the same optimization for `_singlePartitionId`? Can you also benchmark the case with nested filter?



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -30,6 +30,8 @@
 /// Implementations of this interface are assumed not to be stateful. That is, two invocations of
 /// `PartitionFunction.getPartition(value)` with the same value are expected to produce the same
 /// result. Implementations must also be safe for concurrent invocation by multiple threads.
+/// Functions with equal identity may reuse partition ids during broker pruning. Implementations that override
+/// `equals` and `hashCode` must include every setting that affects partition ids, including subclass state.

Review Comment:
   (MAJOR) `BoundedColumnValuePartitionFunction` doesn't have these 2 methods implemented. Is that intentional? If so, we should have a flag to determine whether a partition function can be prepared



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4190949930


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   Agreed that the nested AND/OR case needed more work. I rebased and measured 256/512/1024 candidates with the same two-fork full-prune JMH fixture. The previous lazy path was 6.14/13.60/28.77 µs; the updated PR is 3.34/7.27/15.02 µs. It caches a direct OR's partition-ID union only after a segment misses every branch, so earlier matches still skip unvisited values and later segments avoid traversing the OR tree.
   
   I did not adopt the temporary whole-predicate eager implementation: flattening AND into an ID intersection is incorrect when a segment advertises multiple partitions (e.g. {1,2} should satisfy `id=1 AND id=2` under the current conservative pruner semantics), and eager parsing can throw for literals an earlier branch would have skipped. That temporary version also measured 4.97 vs 3.73 µs for first-hit IN(128) at 256 segments and 10.16 vs 8.31 at 512. The final code keeps those paths lazy; isolated 256-segment first-hit recheck was 3.58 vs 3.62 µs, though some runs were noisy, so I'm not claiming a precise small-case win. The PR description now includes the method and limits. All 30 focused broker cases pass, including a multi-partition segment after OR preparation; exact-head CI is running.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4114884539


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,35 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    List<PreparedPredicate> predicates = new ArrayList<>();
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      int functionHashCode = function.hashCode();
+      PreparedPredicate predicate = null;
+      for (PreparedPredicate candidate : predicates) {

Review Comment:
   I tried the whole-predicate ID-set approach in a temporary full-prune JMH variant (256 segments, one or two functions; 3x1s warmups, 5x1s measurements, two forks on JDK 25). It helps wide filters: homogeneous IN(16) went from 5.70 to 3.10 us/op and OR(4) from 6.43 to 2.73. But it regressed the common EQUALS case from 2.46 to 2.99 (+21%) and early-hit IN(128) from 5.86 to 6.70 (+14%; two functions: 6.49 to 9.56, +47%). Omitting rejected literals also changes the existing short-circuit/error behavior (covered by the invalid-number tests); preserving it needs a fallback path. Given those regressions and the extra path, I kept lazy preparation for this minimal PR. No benchmark source is committed.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +131,28 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    Map<PartitionFunction, PreparedPredicate> predicates = new HashMap<>();

Review Comment:
   Implemented the small query-local list, promoting to HashMap after eight distinct functions (6f6e8de). In the full 256-segment prune benchmark, the one-function cases improved from 4.31 to 2.46 us/op for EQUALS, 8.23 to 5.70 for IN(16), and 4.94 to 2.47 for AND(4); two-function EQUALS improved from 4.63 to 2.99. Homogeneous EQUALS allocation fell from 688 to 552 B/op. A focused test exercises nine distinct functions and verifies ID reuse after promotion. These are local JDK 25 measurements, not a claim about every workload.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4200549913


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   I don't think the multi-partition example demonstrates incorrect query results. It demonstrates different segment selection, and the intersection approach safely prunes more aggressively for a single-valued partition column.
   
   Assuming `partition(id) = id`, consider a segment containing two rows:
   
   ```text
   row 1: id = 1
   row 2: id = 2
   segment partition IDs: {1, 2}
   
   WHERE id = 1 AND id = 2
   ```
   
   The current pruner retains this segment because it independently finds partition 1 and partition 2. Intersecting the predicate ID sets gives `{1} ∩ {2} = ∅`, so it prunes the segment. That is correct: no individual row can satisfy both equalities. Different rows satisfying the two predicates does not make their conjunction satisfiable.
   
   More generally, let `A` and `B` be conservative candidate partition-ID sets for two predicates, computed using the same partition function. Any row satisfying both predicates has one partition-column value `v`, hence one partition ID `p(v)`. That ID must belong to both `A` and `B`, so it must belong to `A ∩ B`. Therefore, for a segment's partition set `S`:
   
   ```text
   S ∩ (A ∩ B) = ∅  =>  the segment cannot contain a matching row
   ```
   
   Multiple partitions within a segment do not invalidate this reasoning. Hash collisions can retain extra candidates, but cannot remove the partition of a matching row. This also extends to nested AND/OR: intersection for AND, union for OR, and all partitions for unsupported predicates, provided the leaf candidate sets are sound.
   
   There would be a concern for a multi-valued partition column, where one row contains `[1, 2]` and separate equality predicates can match different elements. However, [TableConfigUtils explicitly requires partition columns to be single-valued](https://github.com/apache/pinot/blob/abbba34a774b43e7f332dcd6cf684e4940762f8d/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/TableConfigUtils.java#L1861-L1870). Multiple values in one row and multiple partitions in a segment are different cases.
   
   The [new test's final assertion](https://github.com/apache/pinot/blob/abbba34a774b43e7f332dcd6cf684e4940762f8d/pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java#L308-L309) requires retaining the multi-partition segment for this impossible conjunction. That preserves the old pruner's behavior, but does not establish that retaining it is necessary for correctness.
   
   Eager parsing of invalid literals is a separate issue that needs an explicit error-handling policy; it does not make AND intersection unsafe. I think we should evaluate that concern separately from the set-intersection optimization.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4110437293


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -994,6 +1000,10 @@ public static class QueryOptionKey {
         /// Query-level override for `inpredicate.threshold`. Negative means always prune.
         public static final String IN_PREDICATE_PRUNING_THRESHOLD = "inPredicatePruningThreshold";
 
+        /// Set false to use the uncached single-column broker partition pruner. Defaults to true; the cache is only
+        /// used at the configured minimum segment count (default 32). Does not disable partition pruning itself.
+        public static final String ENABLE_PARTITION_PRUNING_CACHE = "enablePartitionPruningCache";

Review Comment:
   Addressed in 0ca0c8641a. The query option is now numeric: `partitionPruningMinSegments`. It overrides the table or cluster threshold, and a negative value disables preparation while keeping partition pruning enabled.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -58,11 +58,20 @@ default List<String> getNames() {
   /// @return Number of possible partitions.
   int getNumPartitions();
 
+  /// Returns the configuration used to construct this function, or null if none is exposed.
   @Nullable
   default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether this function and `other` produce identical partition ids for every input.
+  /// Implementations may opt in using immutable value equality over every output-affecting setting.
+  /// Equality must be reflexive, symmetric and transitive; matching hash codes alone are insufficient.
+  /// The default permits reuse only for the same instance, including for existing plugins with unrelated equality.
+  default boolean canReusePartitionIds(PartitionFunction other) {
+    return this == other;

Review Comment:
   Done in 0ca0c8641a. The default now delegates to `equals(other)`, and the built-ins no longer repeat that override.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    PreparedPredicate predicate = null;
+    PartitionFunction cachedFunction = null;

Review Comment:
   Updated in 0ca0c8641a. The query-local path now keeps a small list with one prepared predicate per distinct compatible partition function. The hash is only a fast precheck; `canReusePartitionIds()` confirms equality before reuse.



##########
pinot-common/src/main/java/org/apache/pinot/common/partition/function/ByteArrayPartitionFunction.java:
##########
@@ -64,6 +64,32 @@ public PartitionIdNormalizer getPartitionIdNormalizer() {
     return _normalizer;
   }
 
+  @Override
+  public boolean canReusePartitionIds(PartitionFunction other) {
+    return equals(other);
+  }
+
+  @Override
+  public boolean equals(Object other) {
+    if (this == other) {
+      return true;
+    }
+    // Subclasses may introduce partitioning state; they must explicitly define their own equality.
+    if (other == null || getClass() != ByteArrayPartitionFunction.class

Review Comment:
   Updated in 0ca0c8641a. All six built-ins now use the standard identity, null, and runtime-class checks in `equals()`.



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -339,6 +339,12 @@ public static class Instance {
   }
 
   public static class Broker {
+    /// Minimum candidate segments for query-local partition-ID caching. Cluster config supports per-table overrides
+    /// by appending `.<tableNameWithType>` to this key. Zero enables caching for any candidate count.

Review Comment:
   Addressed in 0ca0c8641a. The per-table override now lives in `RoutingConfig.partitionPruningMinSegments`; the cluster config supplies only the default. A negative value disables preparation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4110535978


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,35 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    List<PreparedPredicate> predicates = new ArrayList<>();
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      int functionHashCode = function.hashCode();
+      PreparedPredicate predicate = null;
+      for (PreparedPredicate candidate : predicates) {

Review Comment:
   Could we compute the matching partition ID set for the whole predicate once per compatible partition function, then prune segments with a set-overlap check? `EQUALS`/`IN` map values to IDs, `AND` intersects, `OR` unions, and unsupported predicates mean all partitions. Indexing the prepared results by function compatibility would also avoid searching a growing list for every segment.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/manager/BaseBrokerRoutingManager.java:
##########
@@ -209,6 +213,37 @@ public void init(HelixManager helixManager) {
     _propertyStore = helixManager.getHelixPropertyStore();
   }
 
+  @Override
+  public void onChange(Set<String> changedConfigs, Map<String, String> clusterConfigs) {
+    String key = CommonConstants.Broker.CONFIG_OF_PARTITION_PRUNING_MIN_SEGMENTS;
+    int updatedThreshold = CommonConstants.Broker.DEFAULT_PARTITION_PRUNING_MIN_SEGMENTS;
+    String value = clusterConfigs.get(key);
+    if (value != null) {
+      try {
+        updatedThreshold = Integer.parseInt(value);
+      } catch (NumberFormatException e) {
+        LOGGER.warn("Ignoring invalid partition pruning threshold: {}={}", key, value);
+      }
+    }
+    int previousThreshold = _partitionPruningMinSegments;
+    _partitionPruningMinSegments = updatedThreshold;
+    if (previousThreshold == updatedThreshold) {
+      return;
+    }
+    for (String tableNameWithType : _routingEntryMap.keySet()) {

Review Comment:
   Could the pruner read a manager-scoped volatile global threshold when pruning, with table and query overrides taking precedence? A global config update would then be one volatile write instead of synchronously rebuilding every routing entry. This also avoids a route being created with the old value while the update runs.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -58,11 +58,19 @@ default List<String> getNames() {
   /// @return Number of possible partitions.
   int getNumPartitions();
 
+  /// Returns the configuration used to construct this function, or null if none is exposed.
   @Nullable
   default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether this function and `other` produce identical partition ids for every input.
+  /// Implementations that override [#equals(Object)] must include every setting that affects partition ids.
+  /// Matching hash codes alone are insufficient.
+  default boolean canReusePartitionIds(PartitionFunction other) {

Review Comment:
   Could we remove `canReusePartitionIds` and use `equals`/`hashCode` to look up prepared results? Every implementation currently uses the default, which is just `equals`. Keeping a separate SPI method suggests plugins can define another notion of compatibility, but the broker’s hash check can prevent such an override from being used.



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -339,6 +339,10 @@ public static class Instance {
   }
 
   public static class Broker {
+    /// Minimum candidate segments for preparing a partition-pruning predicate. A negative value disables preparation.
+    public static final String CONFIG_OF_PARTITION_PRUNING_MIN_SEGMENTS =

Review Comment:
   Could we call this `partitionPruningPreparationThreshold` consistently? The value controls when predicate preparation starts, rather than when partition pruning starts. I’d suggest `pinot.broker.partition.pruning.preparation.threshold` for the cluster key and `partitionPruningPreparationThreshold` for the table and query options.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4070950166


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {
+    Map<String, String> config = getFunctionConfig();
+    return config == null || config.isEmpty();
+  }
+
+  /// Returns whether partition ids computed by this function can be reused for the non-null `other` function.
+  /// A true result must guarantee identical results for every input value. False is conservative, not proof that the
+  /// functions differ. Implementations with additional output-affecting state must account for it in this method.
+  ///
+  /// The default only permits matching functions with empty exposed configurations, without comparing config contents.
+  /// Configured implementations may override this method to compare their effective settings.
+  default boolean canReusePartitionIds(PartitionFunction other) {
+    return hasEmptyConfig() && other.hasEmptyConfig() && getClass() == other.getClass()

Review Comment:
   Addressed in 115e893c3f. The default now takes the config-content comparison alternative suggested here: an identity fast path, then exact class/name/partition count/resolved normalizer and config contents, with null and empty treated equivalently. The contract explicitly requires reflexivity, symmetry and transitivity. Murmur/Murmur3/FNV compare immutable effective fields instead, and their exposed configs are defensively copied. StarTree Custom already compares its effective expression and normalizer; its companion tests passed against this interface.
   
   One deliberate exception is BoundedColumnValue: it permits self-reuse only. Independently deserialized 100 KB configs made generic map equality cost about 105 us versus 0.69 us for the old EQ path in the exploratory 32-segment screen. The final conservative override keeps the 256-segment large-config controls within about +/-2%. This follows the agreed Apache/StarTree scope; config equality does not claim to protect undisclosed state in arbitrary implementations.
   
   Validation for this commit: 78 focused Apache cases and 46 focused StarTree cases passed. Full PR CI has both integration suites and compatibility checks passing; Unit Test Set 2 is separately failing in the unchanged ZookeeperResourceTest /zk/put HTTP-client path. Resolving this thread for the implemented config-aware alternative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4042063738


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   Updated in d131da5a26: PartitionFunctionKey and its map remain removed; metadata classes are unchanged. The broker now delegates the complete compatibility decision to PartitionFunction.canReusePartitionIds(other), anchored to the first valid segment in each prune call. The default conservatively requires empty exposed configs and matching class/name/count/normalizer. Configured implementations can opt in; StarTree Custom compares its effective expression/count/normalizer in companion PR startreedata/startree-pinot#4428. No cross-query state. Leaving the design preference open for review.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   Updated in d131da5a26: the default interface compatibility check never compares configuration contents, so BoundedColumnValue's large configured maps still take the conservative fallback while reusing the decoded filter. Custom's companion override compares only its effective expression/count/normalizer. The PR description separates historical benchmark results from the new seven-case recheck: the mixed-count case was bimodal in both versions and had a 14.7% slower pooled mean in the new version, so this is not a no-regression claim. All 30 focused Apache test cases passed. Leaving the thread open for review.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4190587624


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -30,6 +30,8 @@
 /// Implementations of this interface are assumed not to be stateful. That is, two invocations of
 /// `PartitionFunction.getPartition(value)` with the same value are expected to produce the same
 /// result. Implementations must also be safe for concurrent invocation by multiple threads.
+/// Functions with equal identity may reuse partition ids during broker pruning. Implementations that override
+/// `equals` and `hashCode` must include every setting that affects partition ids, including subclass state.

Review Comment:
   Intentional. `BoundedColumnValuePartitionFunction` can have a very large configured value array; comparing/hashing it across segments can cost more than direct partition evaluation. I added a default `supportsPartitionIdPreparation()` hook and made Bounded opt out, so the broker uses the original per-segment evaluator for it (096db71a46). The large-config threshold cases pass in the 29-test focused pruner suite.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -30,6 +30,8 @@
 /// Implementations of this interface are assumed not to be stateful. That is, two invocations of
 /// `PartitionFunction.getPartition(value)` with the same value are expected to produce the same
 /// result. Implementations must also be safe for concurrent invocation by multiple threads.
+/// Functions with equal identity may reuse partition ids during broker pruning. Implementations that override

Review Comment:
   Added interface Javadoc in 096db71a46: equality/hash must cover every partition-ID-affecting setting, but the raw config map need not be compared when the parsed fields already capture those settings. Unrelated config entries should not block reuse.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4063350625


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {
+    Map<String, String> config = getFunctionConfig();
+    return config == null || config.isEmpty();
+  }
+
+  /// Returns whether partition ids computed by this function can be reused for the non-null `other` function.
+  /// A true result must guarantee identical results for every input value. False is conservative, not proof that the
+  /// functions differ. Implementations with additional output-affecting state must account for it in this method.
+  ///
+  /// The default only permits matching functions with empty exposed configurations, without comparing config contents.
+  /// Configured implementations may override this method to compare their effective settings.
+  default boolean canReusePartitionIds(PartitionFunction other) {
+    return hasEmptyConfig() && other.hasEmptyConfig() && getClass() == other.getClass()

Review Comment:
   The default is opt-OUT for existing third-party PartitionFunction plugins: any impl whose behaviour is not fully captured by (class, name, numPartitions, normalizer) and that returns null from getFunctionConfig() now silently shares partition ids.
   
   All seven OSS impls (Murmur, Murmur3, Modulo, Fnv, HashCode, ByteArray, BoundedColumnValue) derive every output-affecting field from the function config, so hasEmptyConfig() plus class/count/normalizer really is sufficient for them - I checked each one. But a plugin that loads a lookup table from disk, or takes state from a table-config field it does not echo back in getFunctionConfig(), will be recompiled against this default and start reusing ids across segments that disagree. The javadoc warns implementers, but they have to know to go read it. Safer shape is opt-IN (default false, built-ins override) or config-content equality: getClass() == other.getClass() && Objects.equals(getFunctionConfig(), other.getFunctionConfig()) && getNumPartitions() == ... && getPartitionIdNormalizer() == ... . The second form is strictly more capable too - it would let the BoundedColumnValue 100 KB case share ids, which today is the one workload that gets no benefit (-3.1%, i.e. noise), and a confi
 g == other.getFunctionConfig() identity fast path makes the common interned case free.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {
+    Map<String, String> config = getFunctionConfig();
+    return config == null || config.isEmpty();
+  }
+
+  /// Returns whether partition ids computed by this function can be reused for the non-null `other` function.
+  /// A true result must guarantee identical results for every input value. False is conservative, not proof that the
+  /// functions differ. Implementations with additional output-affecting state must account for it in this method.
+  ///
+  /// The default only permits matching functions with empty exposed configurations, without comparing config contents.
+  /// Configured implementations may override this method to compare their effective settings.
+  default boolean canReusePartitionIds(PartitionFunction other) {

Review Comment:
   This is the load-bearing contract of the whole PR: a wrong 'true' silently drops segments and loses rows with no error anywhere.
   
   The pruner picks the first valid segment's function as a pivot and then fills one shared partition-id cache from whichever segment answers true. For that to be safe, the relation must be symmetric AND transitive over the pivot's compatibility class. The javadoc's phrasing 'ids computed by THIS function can be reused FOR other' reads directional, while the guarantee sentence ('identical results for every input value') is bidirectional. An implementer reading only the first sentence could write a legitimately asymmetric override (e.g. 'other is coarser / a superset'), and the pruner would then cache ids produced by a non-pivot function and hand them to a third segment. Worth stating the required algebra explicitly: canReusePartitionIds must be reflexive, symmetric and transitive, and true implies getPartition(v) is equal for both functions for every v.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -94,6 +98,13 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     if (filterExpression == null) {
       return segments;
     }
+    int numSegments = segments.size();
+    if (numSegments == 0) {
+      return segments;
+    }
+    if (numSegments >= MIN_SEGMENTS_FOR_PREPARATION) {

Review Comment:
   Hardcoded 256-segment cutoff with no escape hatch, and it creates a second evaluator that must stay semantically identical to isPartitionMatch() forever.
   
   Two concerns. (1) No broker config or query option can force the prepared path on or off. The PR's own numbers show the interleaved case is bimodal in both versions and landed 14.7% slower on the pooled mean; if that shows up in a real cluster there is no knob to turn. A `private static final int` read from a CommonConstants broker config, defaulting to 256, would cost nothing. (2) The old loop and isPartitionMatch() stay, so there are now two implementations of the same predicate semantics - including the subtle bits (identifier must equal _partitionColumn, unsupported kinds return true, OR short-circuits before FilterKind.valueOf can throw). The tests do cover both sides via the candidateCounts data provider, but only for some of the cases; a single data-provider-driven equivalence test that runs every fixture at 255 and 256 candidates and asserts the two paths return the same set would pin this down permanently.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +116,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    PreparedPredicate predicate = null;
+    PartitionFunction cachedFunction = null;
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      if (predicate == null) {
+        predicate = new PreparedPredicate(filterExpression);
+        cachedFunction = function;
+      }
+      // Reuse the filter structure even when the functions cannot share partition ids.
+      boolean reusePartitionIds = cachedFunction.canReusePartitionIds(function);
+      if (predicate.matches(partitionInfo.getPartitions(), function, reusePartitionIds)) {

Review Comment:
   Suggestion: once the prepared tree is built and no leaf touched the partition column, the answer is 'all segments' for every remaining segment - you can return `segments` and skip the loop entirely.
   
   For the unrelated-predicate workload the current code still pays a getPartitionFunction() read plus a virtual canReusePartitionIds() call per segment, and the description measures only -7.7% there. If every leaf ended up with _partitionIds == null the predicate is a constant true, so the whole prune degenerates to O(1). Careful with two things: segments with null/INVALID partition info must still be kept (they are, since the answer is 'keep everything'), and the exception behaviour must not change - FilterKind.valueOf on a bad operator has to still throw on the first evaluated segment, which it does because the tree is built before you can know the answer.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +187,82 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private List<Integer> _partitionIds;
+
+    private PreparedPredicate(Expression expression) {
+      _expression = expression;
+    }
+
+    private boolean matches(Set<Integer> partitions, PartitionFunction partitionFunction, boolean reusePartitionIds) {
+      if (_kind == null) {
+        Function function = _expression.getFunctionCall();
+        _kind = FilterKind.valueOf(function.getOperator());
+        _operands = function.getOperands();
+        if (_kind == FilterKind.AND || _kind == FilterKind.OR) {
+          _children = new PreparedPredicate[_operands.size()];
+          for (int i = 0; i < _children.length; i++) {
+            _children[i] = new PreparedPredicate(_operands.get(i));
+          }
+        } else if (_kind == FilterKind.EQUALS || _kind == FilterKind.IN) {
+          Identifier identifier = _operands.get(0).getIdentifier();
+          if (identifier != null && identifier.getName().equals(_partitionColumn)) {
+            _partitionIds = new ArrayList<>(1);

Review Comment:
   new ArrayList<Integer>(1) for an IN list of up to N values: repeated array growth plus a boxed Integer per value, on the path this PR exists to speed up.
   
   _operands.size() - 1 is already known here, so size it (or use an Integer[] with a fill count - the index is always dense and appended at position size(), so an array is a drop-in). Keeping boxed Integer objects is right, since `partitions` is a Set<Integer> and re-boxing per segment would undo part of the win, but growing 1 -> 2 -> 3 -> ... -> 128 does ~8 copies for the 128-value IN benchmark.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +187,82 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private List<Integer> _partitionIds;
+
+    private PreparedPredicate(Expression expression) {
+      _expression = expression;
+    }
+
+    private boolean matches(Set<Integer> partitions, PartitionFunction partitionFunction, boolean reusePartitionIds) {
+      if (_kind == null) {
+        Function function = _expression.getFunctionCall();
+        _kind = FilterKind.valueOf(function.getOperator());
+        _operands = function.getOperands();
+        if (_kind == FilterKind.AND || _kind == FilterKind.OR) {
+          _children = new PreparedPredicate[_operands.size()];
+          for (int i = 0; i < _children.length; i++) {
+            _children[i] = new PreparedPredicate(_operands.get(i));
+          }
+        } else if (_kind == FilterKind.EQUALS || _kind == FilterKind.IN) {
+          Identifier identifier = _operands.get(0).getIdentifier();
+          if (identifier != null && identifier.getName().equals(_partitionColumn)) {
+            _partitionIds = new ArrayList<>(1);
+          }
+        }
+      }
+      switch (_kind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions, partitionFunction, reusePartitionIds)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions, partitionFunction, reusePartitionIds)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds != null) {
+            int numValues = _kind == FilterKind.EQUALS ? 1 : _operands.size() - 1;
+            if (!reusePartitionIds) {

Review Comment:
   Not a finding - recording that I verified this, because it is the part of the diff most likely to be misread as a bug.
   
   numValues maps i in [0, numValues) to _operands.get(i + 1), which matches the original loop over [1, numOperands). Early `return true` on a hit leaves _partitionIds partially filled, but the next segment resumes at exactly i == _partitionIds.size() and appends there, so index i always holds the id for operand i + 1. The `if (!reusePartitionIds)` branch never reads or writes the cache, so an incompatible segment cannot poison or consume the prefix - the testDuplicateInPartitionsAndIncrementalEvaluation fixture is exactly the right test for this. Noting it here because it is the part of the diff most likely to be misread as a bug.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +116,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {

Review Comment:
   Scope: MultiPartitionColumnsSegmentPruner has the same per-segment-per-value hashing loop and gets none of this.
   
   It hashes at lines 142 and 157 in exactly the same shape, so a multi-column-partitioned table with 4096 segments and a 128-value IN still pays the full 100k hashes this PR is about. Fine to leave for a follow-up, but worth saying so in the description so it does not look like the problem is solved table-wide. If a follow-up does extend it, the PreparedPredicate tree would need to be per partition column, which is a good reason to check now that the current shape can be generalised rather than refactored again.



##########
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java:
##########
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.routing.segmentpruner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.partition.PartitionIdNormalizer;
+import org.apache.pinot.segment.spi.partition.metadata.ColumnPartitionMetadata;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises query-local partition ID caching with real metadata initialization and refresh, without ZooKeeper.
+public class SinglePartitionColumnSegmentPrunerTest {
+  private static final String COLUMN = "memberId";
+  private static final String TABLE = "testTable_OFFLINE";
+
+  @Test
+  public void testHashesOnceAcrossDistinctMetadataInstancesPerQuery() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    Set<String> expected = new HashSet<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(i % 8), i % 2 == 0 ? null : Map.of()));
+      if (i % 8 == 3) {
+        expected.add(segment);
+      }
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 1);
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 2, "Computed hashes must not survive a prune call");
+  }
+
+  @Test
+  public void testInterleavedFunctionConfigurationsAndPartitionCounts() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("a", metadata("a", "PrunerCounting", 8, Set.of(3), null));
+    records.put("sameFunction", metadata("sameFunction", "PrunerCounting", 8, Set.of(2), null));
+    records.put("b", metadata("b", "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    records.put("c", metadata("c", "PrunerCounting", 8, Set.of(2), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("e", metadata("e", "PrunerCounting", 16, Set.of(11), null));
+    Set<String> expected = new HashSet<>(Set.of("a", "b", "e"));
+    for (int i = 0; i < 250; i++) {
+      String segment = "tail_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner.prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 4,
+        "Only compatible default functions reuse IDs; configured functions never compare configuration contents");
+    records.put("b", metadata("b", "PrunerCounting", 16, Set.of(11), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), null));
+    expected.add("d");
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 3,
+        "Different partition counts must not reuse partition IDs");
+  }
+
+  @Test
+  public void testDuplicateInPartitionsAndIncrementalEvaluation() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(1), null));
+    records.put("configured", metadata("configured", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("second", metadata("second", "PrunerCounting", 8, Set.of(2), null));
+    records.put("miss", metadata("miss", "PrunerCounting", 8, Set.of(3, 4), null));
+    records.put("repeat", metadata("repeat", "PrunerCounting", 8, Set.of(1, 2), null));
+    Set<String> expected = new HashSet<>(Set.of("first", "configured", "second", "repeat"));
+    for (int i = 0; i < 252; i++) {
+      String segment = "repeat_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(1, 2), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    // The configured segment must not consume or extend the prefix cached by the first segment.
+    assertEquals(pruner.prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()),
+        expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 8);
+    // A configured first segment must not seed IDs for later default-config segments.
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(2), Map.of("offset", "1")));
+    assertEquals(pruner(records).prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()), expected);
+  }
+
+  @Test
+  public void testLargeConfigurationsAndUnrelatedPredicates() throws Exception {
+    String values = "first|" + "x".repeat(100_000);
+    Map<String, String> config = Map.of("columnValues", values, "columnValuesDelimiter", "|");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), config);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), config);
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("first", first, "second", second));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of("first"));
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("value"))), Set.of("first", "second")), Set.of("first", "second"));
+    pruner.refreshSegment("first", metadata("first", "BoundedColumnValue", 3, Set.of(2), config));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of());
+  }
+
+  @Test
+  public void testConfigurationHashCollisionsDoNotReusePartitionIds() throws Exception {

Review Comment:
   This test is vestigial - it guards against a hash collision in a config-hashing scheme the current implementation no longer has.
   
   With the interface-based canReusePartitionIds there is no PartitionFunctionKey and configs are never hashed or compared, so 'Fixture must exercise a configuration hash collision' asserts a property of an earlier design. It also only has 2 candidate segments, which is below MIN_SEGMENTS_FOR_PREPARATION, so it exercises the legacy loop rather than the new code. Harmless, but the name and the assertion message will mislead the next reader. Either drop it or re-point it at whatever the current design actually risks (two configured BoundedColumnValue functions with equal numPartitions and normalizer).



##########
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java:
##########
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.routing.segmentpruner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.partition.PartitionIdNormalizer;
+import org.apache.pinot.segment.spi.partition.metadata.ColumnPartitionMetadata;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises query-local partition ID caching with real metadata initialization and refresh, without ZooKeeper.
+public class SinglePartitionColumnSegmentPrunerTest {
+  private static final String COLUMN = "memberId";
+  private static final String TABLE = "testTable_OFFLINE";
+
+  @Test
+  public void testHashesOnceAcrossDistinctMetadataInstancesPerQuery() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    Set<String> expected = new HashSet<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(i % 8), i % 2 == 0 ? null : Map.of()));
+      if (i % 8 == 3) {
+        expected.add(segment);
+      }
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 1);
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 2, "Computed hashes must not survive a prune call");
+  }
+
+  @Test
+  public void testInterleavedFunctionConfigurationsAndPartitionCounts() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("a", metadata("a", "PrunerCounting", 8, Set.of(3), null));
+    records.put("sameFunction", metadata("sameFunction", "PrunerCounting", 8, Set.of(2), null));
+    records.put("b", metadata("b", "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    records.put("c", metadata("c", "PrunerCounting", 8, Set.of(2), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("e", metadata("e", "PrunerCounting", 16, Set.of(11), null));
+    Set<String> expected = new HashSet<>(Set.of("a", "b", "e"));
+    for (int i = 0; i < 250; i++) {
+      String segment = "tail_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner.prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 4,
+        "Only compatible default functions reuse IDs; configured functions never compare configuration contents");
+    records.put("b", metadata("b", "PrunerCounting", 16, Set.of(11), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), null));
+    expected.add("d");
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 3,
+        "Different partition counts must not reuse partition IDs");
+  }
+
+  @Test
+  public void testDuplicateInPartitionsAndIncrementalEvaluation() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(1), null));
+    records.put("configured", metadata("configured", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("second", metadata("second", "PrunerCounting", 8, Set.of(2), null));
+    records.put("miss", metadata("miss", "PrunerCounting", 8, Set.of(3, 4), null));
+    records.put("repeat", metadata("repeat", "PrunerCounting", 8, Set.of(1, 2), null));
+    Set<String> expected = new HashSet<>(Set.of("first", "configured", "second", "repeat"));
+    for (int i = 0; i < 252; i++) {
+      String segment = "repeat_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(1, 2), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    // The configured segment must not consume or extend the prefix cached by the first segment.
+    assertEquals(pruner.prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()),
+        expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 8);
+    // A configured first segment must not seed IDs for later default-config segments.
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(2), Map.of("offset", "1")));
+    assertEquals(pruner(records).prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()), expected);
+  }
+
+  @Test
+  public void testLargeConfigurationsAndUnrelatedPredicates() throws Exception {
+    String values = "first|" + "x".repeat(100_000);
+    Map<String, String> config = Map.of("columnValues", values, "columnValuesDelimiter", "|");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), config);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), config);
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("first", first, "second", second));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of("first"));
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("value"))), Set.of("first", "second")), Set.of("first", "second"));
+    pruner.refreshSegment("first", metadata("first", "BoundedColumnValue", 3, Set.of(2), config));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of());
+  }
+
+  @Test
+  public void testConfigurationHashCollisionsDoNotReusePartitionIds() throws Exception {
+    Map<String, String> firstConfig = Map.of("columnValues", "Aa|BB", "columnValuesDelimiter", "|");
+    Map<String, String> secondConfig = Map.of("columnValues", "BB|Aa", "columnValuesDelimiter", "|");
+    assertEquals(firstConfig.hashCode(), secondConfig.hashCode(),
+        "Fixture must exercise a configuration hash collision");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), firstConfig);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), secondConfig);
+    Map<String, ZNRecord> records = Map.of("first", first, "second", second);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "Aa")), records.keySet()), records.keySet());
+  }
+
+  @Test
+  public void testMixedFunctionsNormalizersAndFunctionConfig() throws Exception {
+    Map<String, ZNRecord> moduloRecords = new LinkedHashMap<>();
+    moduloRecords.put("positive", metadata("positive", "Modulo", 8, Set.of(7), null));
+    moduloRecords.put("abs", metadata("abs", "Modulo", 8, Set.of(1), Map.of("partitionIdNormalizer", "ABS")));
+    moduloRecords.put("wrongAbs", metadata("wrongAbs", "Modulo", 8, Set.of(7), Map.of("partitionIdNormalizer", "ABS")));
+    Set<String> expected = new HashSet<>(Set.of("positive", "abs"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "positive_" + i;
+      moduloRecords.put(segment, metadata(segment, "Modulo", 8, Set.of(7), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(moduloRecords).prune(request(predicate("EQUALS", "-1")), moduloRecords.keySet()),
+        expected);
+
+    String value = "80ff0102";
+    Map<String, String> rawConfig = Map.of("useRawBytes", "true");
+    int textPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, null).getPartition(value);
+    int rawPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, rawConfig).getPartition(value);
+    assertNotEquals(textPartition, rawPartition, "Fixture must distinguish raw-byte and string hashing");
+    Map<String, ZNRecord> murmurRecords = new LinkedHashMap<>();
+    murmurRecords.put("text", metadata("text", "Murmur", 97, Set.of(textPartition), null));
+    murmurRecords.put("raw", metadata("raw", "Murmur", 97, Set.of(rawPartition), rawConfig));
+    murmurRecords.put("wrongRaw", metadata("wrongRaw", "Murmur", 97, Set.of(textPartition), rawConfig));
+    expected = new HashSet<>(Set.of("text", "raw"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "text_" + i;
+      murmurRecords.put(segment, metadata(segment, "Murmur", 97, Set.of(textPartition), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(murmurRecords).prune(request(predicate("EQUALS", value)), murmurRecords.keySet()),
+        expected);
+
+    Map<String, ZNRecord> lookupRecords = new LinkedHashMap<>();
+    lookupRecords.put("first", metadata("first", "BoundedColumnValue", 3, Set.of(1),
+        Map.of("columnValues", "11|12", "columnValuesDelimiter", "|")));
+    lookupRecords.put("second", metadata("second", "BoundedColumnValue", 3, Set.of(2),
+        Map.of("columnValues", "12|11", "columnValuesDelimiter", "|")));
+    lookupRecords.put("modulo", metadata("modulo", "Modulo", 3, Set.of(2), null));
+    assertEquals(pruner(lookupRecords).prune(request(predicate("EQUALS", "11")), lookupRecords.keySet()),
+        lookupRecords.keySet());
+  }
+
+  @DataProvider
+  public Object[][] candidateCounts() {
+    return new Object[][]{{1}, {2}, {256}};
+  }
+
+  @Test(dataProvider = "candidateCounts")
+  public void testAndOrUnsupportedPredicatesAndLazyInValues(int numSegments) throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < numSegments; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "Modulo", 8, Set.of(1), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    Set<String> segments = records.keySet();
+    Expression invalidValue = predicate("EQUALS", "invalid-number");
+    assertEquals(pruner.prune(request(predicate("IN", "1", "invalid-number")), segments), segments);
+    expectThrows(NumberFormatException.class,
+        () -> pruner.prune(request(predicate("IN", "2", "invalid-number")), segments));
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidValue)), segments), Set.of());
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidValue)), segments), segments);
+    Expression invalidOperator = function("INVALID_OPERATOR");
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidOperator)), segments), segments);
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidOperator)), segments), Set.of());
+    expectThrows(IllegalArgumentException.class,
+        () -> pruner.prune(request(function("OR", predicate("EQUALS", "2"), invalidOperator)), segments));
+    expectThrows(IllegalArgumentException.class, () -> pruner.prune(request(invalidOperator), segments));
+
+    Expression unsupported = predicate("GREATER_THAN", "100");
+    assertEquals(pruner.prune(request(function("AND", predicate("IN", "0", "1"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "2"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("NOT", predicate("EQUALS", "1"))), segments), segments);
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    Expression transformedColumn = function("LOWER", RequestUtils.getIdentifierExpression(COLUMN));
+    assertEquals(pruner.prune(request(function("EQUALS", transformedColumn,
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    BrokerRequest unfilteredRequest = new BrokerRequest();
+    unfilteredRequest.setPinotQuery(new PinotQuery());
+    assertSame(pruner.prune(unfilteredRequest, segments), segments);
+  }
+
+  @Test
+  public void testEmptyCandidatesDoNotEvaluateFilter() throws Exception {
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("one", metadata("one", "Modulo", 8, Set.of(1), null)));
+    assertEquals(pruner.prune(request(function("INVALID_OPERATOR")), Set.of()), Set.of());
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), Set.of()), Set.of());
+  }
+
+  @Test
+  public void testUnknownMetadataIsConservativeAndDoesNotEvaluateFilter() throws Exception {
+    SinglePartitionColumnSegmentPruner pruner = new SinglePartitionColumnSegmentPruner(TABLE, COLUMN);
+    ZNRecord invalid = new ZNRecord("invalid");
+    invalid.setSimpleField(CommonConstants.Segment.PARTITION_METADATA, "invalid-json");
+    pruner.init(null, null, List.of("missing", "empty", "invalid"),
+        Arrays.asList(null, new ZNRecord("empty"), invalid));
+    Set<String> segments = Set.of("missing", "empty", "invalid", "not-initialized");
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), segments), segments);
+    assertEquals(pruner.prune(request(function("INVALID_OPERATOR")), segments), segments);
+  }
+
+  @Test(dataProvider = "candidateCounts")
+  public void testRefreshUsesCurrentPartitionsAndConfiguration(int numSegments) throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < numSegments; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    Set<String> segments = records.keySet();
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, segments), segments);
+    for (String segment : segments) {
+      pruner.refreshSegment(segment, metadata(segment, "PrunerCounting", 8, Set.of(4), null));
+    }
+    assertEquals(pruner.prune(request, segments), Set.of());
+    for (String segment : segments) {
+      pruner.refreshSegment(segment, metadata(segment, "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    }
+    assertEquals(pruner.prune(request, segments), segments);
+    for (String segment : segments) {
+      pruner.refreshSegment(segment, null);
+    }
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), segments), segments);
+  }
+
+  @Test
+  public void testConcurrentQueriesKeepSeparateCachedValues() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "Modulo", 8, Set.of(i % 8), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    ExecutorService executor = Executors.newFixedThreadPool(4);
+    try {
+      List<Callable<Void>> queries = new ArrayList<>();
+      for (int i = 0; i < 32; i++) {
+        int partition = i % 8;
+        Set<String> expected = new HashSet<>();
+        for (int segment = partition; segment < 256; segment += 8) {
+          expected.add("segment_" + segment);
+        }
+        queries.add(() -> {
+          assertEquals(pruner.prune(request(predicate("EQUALS", Integer.toString(partition))), records.keySet()),
+              expected);
+          return null;
+        });
+      }
+      for (Future<Void> future : executor.invokeAll(queries)) {
+        future.get();
+      }
+    } finally {
+      executor.shutdownNow();
+    }
+  }
+
+  private static SinglePartitionColumnSegmentPruner pruner(Map<String, ZNRecord> records) {
+    SinglePartitionColumnSegmentPruner pruner = new SinglePartitionColumnSegmentPruner(TABLE, COLUMN);
+    pruner.init(null, null, new ArrayList<>(records.keySet()), new ArrayList<>(records.values()));
+    return pruner;
+  }
+
+  private static ZNRecord metadata(String segment, String function, int count, Set<Integer> partitions,
+      @Nullable Map<String, String> config) throws Exception {
+    ZNRecord record = new ZNRecord(segment);
+    record.setSimpleField(CommonConstants.Segment.PARTITION_METADATA, new SegmentPartitionMetadata(Map.of(COLUMN,
+        new ColumnPartitionMetadata(function, count, partitions, config))).toJsonString());
+    return record;
+  }
+
+  private static Expression predicate(String operator, String... values) {
+    List<Expression> operands = new ArrayList<>();
+    operands.add(RequestUtils.getIdentifierExpression(COLUMN));
+    for (String value : values) {
+      operands.add(RequestUtils.getLiteralExpression(value));
+    }
+    return RequestUtils.getFunctionExpression(operator, operands);
+  }
+
+  private static Expression function(String operator, Expression... operands) {
+    return RequestUtils.getFunctionExpression(operator, List.of(operands));
+  }
+
+  private static BrokerRequest request(Expression filter) {
+    PinotQuery pinotQuery = new PinotQuery();
+    pinotQuery.setFilterExpression(filter);
+    BrokerRequest brokerRequest = new BrokerRequest();
+    brokerRequest.setPinotQuery(pinotQuery);
+    return brokerRequest;
+  }
+
+  /// Stateless partition function with observable hash calls.
+  public static class CountingPartitionFunction implements PartitionFunction {
+    private static final long serialVersionUID = 1L;
+    private static final AtomicInteger CALLS = new AtomicInteger();

Review Comment:
   The exact-call-count assertions on this static counter are safe today, but only because of the repo's surefire settings - worth a comment in the test so it is not lost.
   
   Three tests reset CALLS and assert exact values. The root pom sets <parallel>false</parallel> and reuseForks=false, so methods run sequentially and this class gets its own JVM. If the module ever enabled TestNG method-level parallelism these would interleave and flake. Also note CountingPartitionFunction is public and in org.apache.pinot, so PartitionFunctionFactory's classpath scan registers 'PrunerCounting' globally for every test in this fork - intentional and needed here, just a global side effect to be aware of.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {

Review Comment:
   hasEmptyConfig() becomes permanent public SPI surface, but it exists only to support the default canReusePartitionIds. A private interface method would do the same job without the commitment.
   
   Java 9+ private interface methods are callable on any receiver from inside the interface body, so `private boolean hasEmptyConfig()` still lets the default impl call other.hasEmptyConfig(). SPI method names are forever in Pinot; I would rather not add one whose only caller is two lines below unless overriders are expected to reuse it. If it is meant as a public helper for overriders, say so in the javadoc. The @JsonIgnore is correct and necessary, by the way - Jackson would otherwise expose it as an 'emptyConfig' property on serialised partition metadata.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +116,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    PreparedPredicate predicate = null;
+    PartitionFunction cachedFunction = null;
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      if (predicate == null) {
+        predicate = new PreparedPredicate(filterExpression);
+        cachedFunction = function;
+      }
+      // Reuse the filter structure even when the functions cannot share partition ids.
+      boolean reusePartitionIds = cachedFunction.canReusePartitionIds(function);

Review Comment:
   The pivot is whichever segment HashSet iteration happens to visit first, and for any configured partition function the first call is a self-comparison that returns false - so the optimization silently disables itself for the whole query.
   
   cachedFunction is pinned to the first valid segment, then compared against itself on that same iteration. With the default canReusePartitionIds, f.canReusePartitionIds(f) is FALSE whenever f has a non-empty functionConfig, because hasEmptyConfig() fails on both sides. So the default impl is not reflexive, which is surprising for a relation named 'canReuse...', and it means a configured table gets zero id reuse - matching the BOUNDED_100KB rows in the description. Two follow-ons: (a) worth a `function == cachedFunction` identity short-circuit before the virtual call, both for speed and so self-comparison is trivially true; (b) one outlier segment arriving first (a stale numPartitions after a table-config change, say) disables reuse for the other N-1 segments. If that matters, re-pivot when the pivot yields no reuse for the first few segments, or key a small map by function instead of one pivot.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -94,6 +98,13 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     if (filterExpression == null) {
       return segments;
     }
+    int numSegments = segments.size();
+    if (numSegments == 0) {

Review Comment:
   This branch is unreachable in production and it changes what the method returns: the caller's set instead of a fresh one.
   
   BaseBrokerRoutingManager.selectThenPrune() guards the whole pruner chain with `if (!selectedSegments.isEmpty())`, so prune() is never called with an empty set on the query path. The old code already never evaluated the filter for an empty input (the loop body just did not run), so the branch buys nothing behaviourally. What it does change is aliasing: getPrunedSegments() documents that 'the pruners return a new set rather than editing the one they are handed'. The filterExpression == null branch above already breaks that, so this is not new, but I would drop the branch rather than add a second exception to a convention a neighbouring comment relies on.



##########
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java:
##########
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.routing.segmentpruner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.partition.PartitionIdNormalizer;
+import org.apache.pinot.segment.spi.partition.metadata.ColumnPartitionMetadata;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises query-local partition ID caching with real metadata initialization and refresh, without ZooKeeper.
+public class SinglePartitionColumnSegmentPrunerTest {
+  private static final String COLUMN = "memberId";
+  private static final String TABLE = "testTable_OFFLINE";
+
+  @Test
+  public void testHashesOnceAcrossDistinctMetadataInstancesPerQuery() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    Set<String> expected = new HashSet<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(i % 8), i % 2 == 0 ? null : Map.of()));
+      if (i % 8 == 3) {
+        expected.add(segment);
+      }
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 1);
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 2, "Computed hashes must not survive a prune call");
+  }
+
+  @Test
+  public void testInterleavedFunctionConfigurationsAndPartitionCounts() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("a", metadata("a", "PrunerCounting", 8, Set.of(3), null));
+    records.put("sameFunction", metadata("sameFunction", "PrunerCounting", 8, Set.of(2), null));
+    records.put("b", metadata("b", "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    records.put("c", metadata("c", "PrunerCounting", 8, Set.of(2), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("e", metadata("e", "PrunerCounting", 16, Set.of(11), null));
+    Set<String> expected = new HashSet<>(Set.of("a", "b", "e"));
+    for (int i = 0; i < 250; i++) {
+      String segment = "tail_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner.prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 4,
+        "Only compatible default functions reuse IDs; configured functions never compare configuration contents");
+    records.put("b", metadata("b", "PrunerCounting", 16, Set.of(11), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), null));
+    expected.add("d");
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 3,
+        "Different partition counts must not reuse partition IDs");
+  }
+
+  @Test
+  public void testDuplicateInPartitionsAndIncrementalEvaluation() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(1), null));
+    records.put("configured", metadata("configured", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("second", metadata("second", "PrunerCounting", 8, Set.of(2), null));
+    records.put("miss", metadata("miss", "PrunerCounting", 8, Set.of(3, 4), null));
+    records.put("repeat", metadata("repeat", "PrunerCounting", 8, Set.of(1, 2), null));
+    Set<String> expected = new HashSet<>(Set.of("first", "configured", "second", "repeat"));
+    for (int i = 0; i < 252; i++) {
+      String segment = "repeat_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(1, 2), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    // The configured segment must not consume or extend the prefix cached by the first segment.
+    assertEquals(pruner.prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()),
+        expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 8);
+    // A configured first segment must not seed IDs for later default-config segments.
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(2), Map.of("offset", "1")));
+    assertEquals(pruner(records).prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()), expected);
+  }
+
+  @Test
+  public void testLargeConfigurationsAndUnrelatedPredicates() throws Exception {
+    String values = "first|" + "x".repeat(100_000);
+    Map<String, String> config = Map.of("columnValues", values, "columnValuesDelimiter", "|");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), config);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), config);
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("first", first, "second", second));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of("first"));
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("value"))), Set.of("first", "second")), Set.of("first", "second"));
+    pruner.refreshSegment("first", metadata("first", "BoundedColumnValue", 3, Set.of(2), config));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of());
+  }
+
+  @Test
+  public void testConfigurationHashCollisionsDoNotReusePartitionIds() throws Exception {
+    Map<String, String> firstConfig = Map.of("columnValues", "Aa|BB", "columnValuesDelimiter", "|");
+    Map<String, String> secondConfig = Map.of("columnValues", "BB|Aa", "columnValuesDelimiter", "|");
+    assertEquals(firstConfig.hashCode(), secondConfig.hashCode(),
+        "Fixture must exercise a configuration hash collision");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), firstConfig);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), secondConfig);
+    Map<String, ZNRecord> records = Map.of("first", first, "second", second);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "Aa")), records.keySet()), records.keySet());
+  }
+
+  @Test
+  public void testMixedFunctionsNormalizersAndFunctionConfig() throws Exception {
+    Map<String, ZNRecord> moduloRecords = new LinkedHashMap<>();
+    moduloRecords.put("positive", metadata("positive", "Modulo", 8, Set.of(7), null));
+    moduloRecords.put("abs", metadata("abs", "Modulo", 8, Set.of(1), Map.of("partitionIdNormalizer", "ABS")));
+    moduloRecords.put("wrongAbs", metadata("wrongAbs", "Modulo", 8, Set.of(7), Map.of("partitionIdNormalizer", "ABS")));
+    Set<String> expected = new HashSet<>(Set.of("positive", "abs"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "positive_" + i;
+      moduloRecords.put(segment, metadata(segment, "Modulo", 8, Set.of(7), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(moduloRecords).prune(request(predicate("EQUALS", "-1")), moduloRecords.keySet()),
+        expected);
+
+    String value = "80ff0102";
+    Map<String, String> rawConfig = Map.of("useRawBytes", "true");
+    int textPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, null).getPartition(value);
+    int rawPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, rawConfig).getPartition(value);
+    assertNotEquals(textPartition, rawPartition, "Fixture must distinguish raw-byte and string hashing");
+    Map<String, ZNRecord> murmurRecords = new LinkedHashMap<>();
+    murmurRecords.put("text", metadata("text", "Murmur", 97, Set.of(textPartition), null));
+    murmurRecords.put("raw", metadata("raw", "Murmur", 97, Set.of(rawPartition), rawConfig));
+    murmurRecords.put("wrongRaw", metadata("wrongRaw", "Murmur", 97, Set.of(textPartition), rawConfig));
+    expected = new HashSet<>(Set.of("text", "raw"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "text_" + i;
+      murmurRecords.put(segment, metadata(segment, "Murmur", 97, Set.of(textPartition), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(murmurRecords).prune(request(predicate("EQUALS", value)), murmurRecords.keySet()),
+        expected);
+
+    Map<String, ZNRecord> lookupRecords = new LinkedHashMap<>();
+    lookupRecords.put("first", metadata("first", "BoundedColumnValue", 3, Set.of(1),
+        Map.of("columnValues", "11|12", "columnValuesDelimiter", "|")));
+    lookupRecords.put("second", metadata("second", "BoundedColumnValue", 3, Set.of(2),
+        Map.of("columnValues", "12|11", "columnValuesDelimiter", "|")));
+    lookupRecords.put("modulo", metadata("modulo", "Modulo", 3, Set.of(2), null));
+    assertEquals(pruner(lookupRecords).prune(request(predicate("EQUALS", "11")), lookupRecords.keySet()),
+        lookupRecords.keySet());
+  }
+
+  @DataProvider
+  public Object[][] candidateCounts() {
+    return new Object[][]{{1}, {2}, {256}};
+  }
+
+  @Test(dataProvider = "candidateCounts")
+  public void testAndOrUnsupportedPredicatesAndLazyInValues(int numSegments) throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < numSegments; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "Modulo", 8, Set.of(1), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    Set<String> segments = records.keySet();
+    Expression invalidValue = predicate("EQUALS", "invalid-number");
+    assertEquals(pruner.prune(request(predicate("IN", "1", "invalid-number")), segments), segments);
+    expectThrows(NumberFormatException.class,
+        () -> pruner.prune(request(predicate("IN", "2", "invalid-number")), segments));
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidValue)), segments), Set.of());
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidValue)), segments), segments);
+    Expression invalidOperator = function("INVALID_OPERATOR");
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidOperator)), segments), segments);
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidOperator)), segments), Set.of());
+    expectThrows(IllegalArgumentException.class,
+        () -> pruner.prune(request(function("OR", predicate("EQUALS", "2"), invalidOperator)), segments));
+    expectThrows(IllegalArgumentException.class, () -> pruner.prune(request(invalidOperator), segments));
+
+    Expression unsupported = predicate("GREATER_THAN", "100");
+    assertEquals(pruner.prune(request(function("AND", predicate("IN", "0", "1"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "2"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("NOT", predicate("EQUALS", "1"))), segments), segments);
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    Expression transformedColumn = function("LOWER", RequestUtils.getIdentifierExpression(COLUMN));
+    assertEquals(pruner.prune(request(function("EQUALS", transformedColumn,
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    BrokerRequest unfilteredRequest = new BrokerRequest();
+    unfilteredRequest.setPinotQuery(new PinotQuery());
+    assertSame(pruner.prune(unfilteredRequest, segments), segments);
+  }
+
+  @Test
+  public void testEmptyCandidatesDoNotEvaluateFilter() throws Exception {
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("one", metadata("one", "Modulo", 8, Set.of(1), null)));
+    assertEquals(pruner.prune(request(function("INVALID_OPERATOR")), Set.of()), Set.of());
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), Set.of()), Set.of());
+  }
+
+  @Test
+  public void testUnknownMetadataIsConservativeAndDoesNotEvaluateFilter() throws Exception {

Review Comment:
   Test gap: the all-invalid-metadata case only runs on the legacy path (4 candidates), yet it is the one case where the prepared path never builds the predicate at all.
   
   In pruneWithPreparedPredicate the PreparedPredicate is constructed only when the first VALID partition info is found, so when every segment is null/INVALID the filter is never decoded and FilterKind.valueOf never throws. That is the same conservative behaviour as the original short-circuit, but it is a distinct code path and this test does not reach it. Padding the fixture past 256 candidates (or adding it to the candidateCounts data provider) would cover it. Same thought for testLargeConfigurationsAndUnrelatedPredicates, which has 2 segments and therefore never touches the prepared evaluator it is named after.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4110332215


##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -994,6 +1000,10 @@ public static class QueryOptionKey {
         /// Query-level override for `inpredicate.threshold`. Negative means always prune.
         public static final String IN_PREDICATE_PRUNING_THRESHOLD = "inPredicatePruningThreshold";
 
+        /// Set false to use the uncached single-column broker partition pruner. Defaults to true; the cache is only
+        /// used at the configured minimum segment count (default 32). Does not disable partition pruning itself.
+        public static final String ENABLE_PARTITION_PRUNING_CACHE = "enablePartitionPruningCache";

Review Comment:
   To keep the behavior consistent, suggest making the table option also configuring the min segments to trigger the pruning preparation. We can use negative value to turn it off



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -58,11 +58,20 @@ default List<String> getNames() {
   /// @return Number of possible partitions.
   int getNumPartitions();
 
+  /// Returns the configuration used to construct this function, or null if none is exposed.
   @Nullable
   default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether this function and `other` produce identical partition ids for every input.
+  /// Implementations may opt in using immutable value equality over every output-affecting setting.
+  /// Equality must be reflexive, symmetric and transitive; matching hash codes alone are insufficient.
+  /// The default permits reuse only for the same instance, including for existing plugins with unrelated equality.
+  default boolean canReusePartitionIds(PartitionFunction other) {
+    return this == other;

Review Comment:
   Return `equals(other)` by default?



##########
pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java:
##########
@@ -339,6 +339,12 @@ public static class Instance {
   }
 
   public static class Broker {
+    /// Minimum candidate segments for query-local partition-ID caching. Cluster config supports per-table overrides
+    /// by appending `.<tableNameWithType>` to this key. Zero enables caching for any candidate count.

Review Comment:
   This is not the correct way to do table level override. Table level override should be done through the table config. We can use negative value to turn this off



##########
pinot-common/src/main/java/org/apache/pinot/common/partition/function/ByteArrayPartitionFunction.java:
##########
@@ -64,6 +64,32 @@ public PartitionIdNormalizer getPartitionIdNormalizer() {
     return _normalizer;
   }
 
+  @Override
+  public boolean canReusePartitionIds(PartitionFunction other) {
+    return equals(other);
+  }
+
+  @Override
+  public boolean equals(Object other) {
+    if (this == other) {
+      return true;
+    }
+    // Subclasses may introduce partitioning state; they must explicitly define their own equality.
+    if (other == null || getClass() != ByteArrayPartitionFunction.class

Review Comment:
   I don't think this is the standard way of doing `equals`



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    PreparedPredicate predicate = null;
+    PartitionFunction cachedFunction = null;

Review Comment:
   This degrades when there are multiple functions. Consider using a List (given there are usually very few functions)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19539:
URL: https://github.com/apache/pinot/pull/19539#issuecomment-5649272718

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19539?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 57.82%. Comparing base ([`5771d6a`](https://app.codecov.io/gh/apache/pinot/commit/5771d6acea60cd72738116835111cf7c49965373?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8b15fbc`](https://app.codecov.io/gh/apache/pinot/commit/8b15fbc6561e39dca7c17d14aa172657a63913db?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   > :exclamation:  There is a different number of reports uploaded between BASE (5771d6a) and HEAD (8b15fbc). Click for more details.
   > 
   > <details><summary>HEAD has 14 uploads less than BASE</summary>
   >
   >| Flag | BASE (5771d6a) | HEAD (8b15fbc) |
   >|------|------|------|
   >|unittests|2|1|
   >|java-25|6|3|
   >|temurin|6|3|
   >|lane-a|2|1|
   >|integration|4|2|
   >|lane-b|2|1|
   >|integration1|2|0|
   >|unittests2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19539      +/-   ##
   ============================================
   - Coverage     67.72%   57.82%   -9.90%     
   + Complexity     1450        1    -1449     
   ============================================
     Files          3490     2688     -802     
     Lines        225032   164362   -60670     
     Branches      35527    26686    -8841     
   ============================================
   - Hits         152393    95038   -57355     
   - Misses        60612    61295     +683     
   + Partials      12027     8029    -3998     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.82% <ø> (-9.90%)` | :arrow_down: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.82% <ø> (-9.90%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.82% <ø> (-9.90%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.82% <ø> (+0.01%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19539/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19539?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4042063613


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {

Review Comment:
   Revisited in e7b49588e3: removed the duplicate isPartitionMatch() implementation in favor of one PreparedPredicate evaluator, following the later suggestion to use one path for singleton and multi-segment inputs. The earlier statement that this method was unchanged was incorrect.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   Addressed in e7b49588e3. Partition-function identity now belongs to SegmentPartitionInfo: it creates an immutable snapshot and canonicalizes equal keys when metadata is loaded/refreshed. The query uses those shared keys in an IdentityHashMap, so PartitionFunctionKey/CachedPartitionFunctionKey/PartitionFunctionLookup are no longer needed in the pruner. Segment-specific partition sets remain separate and are checked for each segment.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {

Review Comment:
   Corrected in e7b49588e3. Each PreparedPredicate initializes its own final fields once in the constructor. AND/OR children are constructed only when visited, preserving short-circuit behavior for invalid later expressions. The previous revision assigned final fields twice and failed compilation before integration tests ran. JDK 25 broker compilation and all 26 focused broker tests now pass, including short-circuited invalid operators and values.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   Addressed in e7b49588e3: empty candidates return immediately, and singleton/multi-segment inputs now use the same matcher/evaluator. Removed the duplicate uncached method. This accepts matcher setup for singleton inputs in exchange for one evaluation path; the old singleton benchmark results describe the earlier implementation, not this revision.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;

Review Comment:
   Addressed in e7b49588e3 using an IntSet. It retains distinct computed partition IDs and a count of evaluated literals. Each segment first checks the cached IDs, then hashes additional literals only until a match, preserving lazy IN evaluation. Added coverage for repeated partition IDs and extending the evaluated prefix across segments; existing lazy-error tests still pass.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   The earlier hash precheck did not address the equal-configuration case you described. Fixed in e7b49588e3: SegmentPartitionInfo canonicalizes immutable function keys using full equality during metadata construction/refresh. Query lookups use IdentityHashMap and do not compare or hash configuration maps. The interner uses weak references so unused keys can be reclaimed. Added regressions for independently loaded 100 KB BoundedColumnValue configs sharing identity, distinct configs with colliding hashes, and metadata refresh. All 26 focused broker tests pass; no new timing benchmark is claimed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4067648333


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): The wrapper/key and its map have been removed. Compatibility now belongs to PartitionFunction; metadata objects remain unchanged and the cache is local to a single prune call. This addresses the wrapper concern without expanding SegmentPartitionInfo.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): The current design deliberately retains the original evaluator below 256 candidates to avoid preparation overhead. The prepared path is lazy, and the expanded shared fixtures now exercise 255/256 candidates plus a forced uncached path through enablePartitionPruningCache=false. This addresses the coverage and operational fallback concerns, but not the preference for maintaining a single evaluator. Leaving that design choice open rather than claiming it is resolved.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Keeping decoding lazy is necessary for the current evaluator's short-circuit behavior: an invalid operator or literal in an unvisited AND/OR/IN branch must not start throwing. The shared fixtures exercise this at 255 and 256 candidates and with caching disabled. Constructor preparation of the whole tree would violate those cases; the lazy fields and their purpose are retained.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Kept the ordered visited-prefix list intentionally. Deduplicating eagerly into an IntSet would evaluate remaining literals after an earlier match, changing lazy-error behavior and adding work for early-matching long IN filters. The cache computes each visited literal once, while incompatible functions neither read nor extend it. Deduplication can be a separate measured optimization without complicating this query-local change.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): The full config-map comparison has been removed. The interface default checks only config emptiness (or instance identity) and small function attributes; BoundedColumnValue's independently deserialized large configs conservatively fall back. The large-config/unrelated-column fixture now reaches 255/256 candidates and the cache-disabled path. No metadata interning or per-segment scan of the 100 KB string is introduced.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {
+    Map<String, String> config = getFunctionConfig();
+    return config == null || config.isEmpty();
+  }
+
+  /// Returns whether partition ids computed by this function can be reused for the non-null `other` function.
+  /// A true result must guarantee identical results for every input value. False is conservative, not proof that the
+  /// functions differ. Implementations with additional output-affecting state must account for it in this method.
+  ///
+  /// The default only permits matching functions with empty exposed configurations, without comparing config contents.
+  /// Configured implementations may override this method to compare their effective settings.
+  default boolean canReusePartitionIds(PartitionFunction other) {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Clarified the SPI contract explicitly: reflexive, symmetric, transitive, and equal partition IDs for every input; coarser/superset relations are not sufficient. Added the same-instance fast path, including configured instances, and extended the compatibility test for reflexivity and a three-instance compatible group. Callers may safely populate the cache from any compatible member.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4113546451


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,35 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    List<PreparedPredicate> predicates = new ArrayList<>();
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      int functionHashCode = function.hashCode();
+      PreparedPredicate predicate = null;
+      for (PreparedPredicate candidate : predicates) {

Review Comment:
   Could we resolve the whole predicate to candidate partition IDs once per distinct partition function? `EQUALS`/`IN` contribute IDs, `AND` intersects, `OR` unions, and an unrelated predicate acts as a wildcard. A literal rejected as invalid by the partition function can be omitted from its ID set; I’d catch only expected invalid-input errors. This would leave one ID-overlap check per segment instead of walking the predicate tree for every segment. Please benchmark the common one-function case, including simple filters and wider `AND`/`OR` filters, against the current lazy approach.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +131,28 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    Map<PartitionFunction, PreparedPredicate> predicates = new HashMap<>();

Review Comment:
   Most queries appear to encounter only one or a few distinct partition functions. Could we keep prepared results in a small list and promote to a `HashMap` after several distinct functions, perhaps eight? That avoids hashing a newly constructed function for every segment in the common case while retaining fast lookup when functions vary widely. A lookup-only benchmark favors the list for one function, though we should measure the full pruning path before fixing the cutoff.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4201480263


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +223,106 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private boolean _isPartitionPredicate;
+    private Integer _singlePartitionId;
+    private IntSet _partitionIds;
+    private int _numEvaluatedValues;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private boolean matches(Set<Integer> partitions) {

Review Comment:
   Agreed; `850bad9f22` uses whole-predicate eager preparation, not the previous lazy/OR-on-miss variant. This supersedes my earlier replies that kept preparation lazy. The candidate IDs are prepared once per equivalent partition function per prune call, then reused for segment overlap checks. AND intersects, OR unions, and unsupported predicates are unrestricted. Partition columns are validated as single-valued, so the intersection is safe even when a segment contains multiple partitions. The cached singleton-ID fast path is retained.
   
   Invalid-input handling is separate: if eager preparation throws, that function group uses the original per-segment evaluator for the rest of the call. We do not silently discard invalid literals; original short-circuit/error behavior is preserved. Below the preparation threshold, and for functions that opt out, the original evaluator still applies.
   
   I reran the published implementation against `abbba34a77` (previous cached implementation), including 256, 512, 1024, 4096, and 16384 segments. The preparation-disabled control is the original per-segment evaluator at the current revision, not a separately built master. Same harness/dependencies, freshly compiled pruner revisions, JDK 25.0.4.1, JMH 1.37, one thread, two forks, 3 × 1-second warmups / 5 × 1-second measurements per fork, average time, `-prof gc`. Independently constructed equal 64-way Modulo functions, one partition per segment. Every fixture checks selected segments before timing.
   
   Nested filter: `(id = 7 OR id IN (0, ..., 15)) AND otherColumn = 1`; the unrelated column acts as a wildcard. First-hit IN(128) uses partition 0 for every segment. Scores below are µs per full prune call, mean ± JMH-reported 99.9% error. The interpretation column uses a conservative twice-relative-error heuristic, not a formal paired significance test.
   
   <details>
   <summary>Full main-run latency matrix</summary>
   
   | Filter | Segments | Preparation disabled (µs/op) | Before cached (µs/op) | Eager (µs/op) | Eager vs before | Interpretation |
   |---|---:|---:|---:|---:|---:|---|
   | EQUALS | 256 | 3.81 ± 1.15 | 1.76 ± 0.10 | 1.63 ± 0.03 | -7.3% | Inconclusive |
   | EQUALS | 512 | 6.91 ± 0.20 | 3.71 ± 0.26 | 3.22 ± 0.29 | -13.2% | Inconclusive |
   | EQUALS | 1,024 | 16.26 ± 0.86 | 9.35 ± 1.94 | 7.55 ± 0.24 | -19.3% | Inconclusive |
   | EQUALS | 4,096 | 63.43 ± 1.34 | 36.93 ± 5.26 | 29.97 ± 1.62 | -18.8% | Inconclusive |
   | EQUALS | 16,384 | 323.47 ± 4.93 | 152.53 ± 2.13 | 140.52 ± 15.72 | -7.9% | Inconclusive |
   | IN(16) | 256 | 26.17 ± 0.27 | 2.58 ± 0.03 | 3.61 ± 0.34 | +39.7% | Likely regression |
   | IN(16) | 512 | 76.10 ± 0.73 | 5.83 ± 0.18 | 5.50 ± 0.28 | -5.6% | Inconclusive |
   | IN(16) | 1,024 | 147.08 ± 9.87 | 12.69 ± 0.30 | 12.02 ± 1.09 | -5.2% | Inconclusive |
   | IN(16) | 4,096 | 614.52 ± 42.12 | 50.59 ± 1.83 | 55.23 ± 33.58 | +9.2% | Inconclusive |
   | IN(16) | 16,384 | 1954.26 ± 29.93 | 233.58 ± 19.48 | 211.82 ± 21.19 | -9.3% | Inconclusive |
   | Nested AND/OR | 256 | 32.72 ± 1.54 | 3.69 ± 0.12 | 3.76 ± 0.56 | +1.9% | Inconclusive |
   | Nested AND/OR | 512 | 92.21 ± 2.34 | 7.95 ± 0.79 | 5.33 ± 0.15 | -33.0% | Likely improvement |
   | Nested AND/OR | 1,024 | 187.26 ± 11.98 | 15.51 ± 0.34 | 11.58 ± 0.71 | -25.3% | Likely improvement |
   | Nested AND/OR | 4,096 | 713.18 ± 7.91 | 69.60 ± 3.87 | 45.00 ± 2.10 | -35.3% | Likely improvement |
   | Nested AND/OR | 16,384 | 2454.42 ± 160.26 | 307.59 ± 7.72 | 204.75 ± 9.17 | -33.4% | Likely improvement |
   | IN(128), first hit | 256 | 5.50 ± 0.31 | 10.09 ± 26.19 | 5.27 ± 1.21 | -47.8% | Inconclusive |
   | IN(128), first hit | 512 | 11.82 ± 0.36 | 8.50 ± 0.74 | 9.36 ± 1.26 | +10.1% | Inconclusive |
   | IN(128), first hit | 1,024 | 24.80 ± 1.20 | 17.72 ± 0.69 | 17.40 ± 0.41 | -1.8% | Inconclusive |
   | IN(128), first hit | 4,096 | 95.30 ± 1.02 | 78.98 ± 10.39 | 63.89 ± 1.28 | -19.1% | Inconclusive |
   | IN(128), first hit | 16,384 | 419.06 ± 8.90 | 371.91 ± 41.33 | 282.40 ± 11.20 | -24.1% | Likely improvement |
   
   </details>
   
   Two noisy controls and the small-case cost were checked with complete before/eager reruns at both 256 and 4096 segments, using fresh forks and the same settings. All four paired repeats are included; the original matrix is retained above:
   
   | Filter | Segments | Before cached (µs/op) | Eager (µs/op) | Change |
   |---|---:|---:|---:|---:|
   | IN(16) | 256 | 2.60 ± 0.04 | 3.23 ± 0.04 | +23.9% |
   | IN(16) | 4,096 | 50.24 ± 2.53 | 45.00 ± 2.36 | -10.4% |
   | IN(128), first hit | 256 | 3.90 ± 0.29 | 4.63 ± 0.07 | +18.6% |
   | IN(128), first hit | 4,096 | 71.22 ± 6.94 | 66.90 ± 5.30 | -6.1% |
   
   Nested pruning improves about 33%, 25%, 35%, and 33% at 512/1024/4096/16384 segments versus the previous cached implementation. The 256 nested cell is inconclusive in this rerun. The stable 256 repeats show a small eager cost: +0.62 µs for IN(16), +0.73 µs for first-hit IN(128). Several cheap/large controls still have wide intervals, so I am not claiming universal no-regression.
   
   Where the savings come from: preparation disabled recomputes literal IDs and visits the predicate for each segment. The previous cache already avoids most repeated ID computation; eager additionally collapses the full predicate once and removes per-segment tree traversal. At 4096 segments, nested pruning is 713.18 µs without preparation, 69.60 µs before cached, and 45.00 µs eager. Allocation falls by about 1192 B/prune for nested at 1024+ segments versus before cached; first-hit IN adds about 3464 B/prune because every literal is prepared.
   
   The PR description is being updated to these results and tradeoffs. Apache CI is green on `850bad9f22` (13 checks); this rerun changes no production code or tests, and no benchmark source is committed. These are synthetic broker-pruner measurements, not end-to-end customer QPS.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4201485779


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +125,35 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    List<PreparedPredicate> predicates = new ArrayList<>();
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      int functionHashCode = function.hashCode();
+      PreparedPredicate predicate = null;
+      for (PreparedPredicate candidate : predicates) {

Review Comment:
   Implemented in `850bad9f22`: the prepared path now resolves the whole supported predicate eagerly once per equivalent partition function per prune call, then uses segment overlap checks. AND intersects, OR unions, unrelated predicates are unrestricted, and singleton results retain the fast membership path. Single-valued partition-column validation makes the intersection safe; preparation errors fall back to the original evaluator rather than discarding invalid values. This supersedes my earlier replies that kept preparation lazy.
   
   The [fresh benchmark rerun](https://github.com/apache/pinot/pull/19539#discussion_r4201480263) compares the previous cached implementation and preparation-disabled control at 256/512/1024/4096/16384 segments, including allocation measurements and paired rechecks. Nested filters improve about 25–35% at 512+ segments versus before cached; the 256 repeats show +0.62 µs for IN(16) and +0.73 µs for first-hit IN. The PR description includes the gains, small-case costs, and inconclusive controls. No production code or benchmark source is added in this rerun; all 13 Apache CI checks pass.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4067648944


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {
+    Map<String, String> config = getFunctionConfig();
+    return config == null || config.isEmpty();
+  }
+
+  /// Returns whether partition ids computed by this function can be reused for the non-null `other` function.
+  /// A true result must guarantee identical results for every input value. False is conservative, not proof that the
+  /// functions differ. Implementations with additional output-affecting state must account for it in this method.
+  ///
+  /// The default only permits matching functions with empty exposed configurations, without comparing config contents.
+  /// Configured implementations may override this method to compare their effective settings.
+  default boolean canReusePartitionIds(PartitionFunction other) {
+    return hasEmptyConfig() && other.hasEmptyConfig() && getClass() == other.getClass()

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): The default now states its assumptions explicitly. For this change we are intentionally retaining the agreed Apache/StarTree implementation scope: the audited OSS functions expose output-affecting configuration, and StarTree Custom overrides compatibility using its effective settings. Config-map equality would not protect hidden state when both maps are null, and it reintroduces the independently deserialized 100 KB comparison cost discussed above. An opt-in default with built-in overrides is the safer choice if arbitrary plugins must be supported, but broadens this agreed implementation; leaving that policy decision open for maintainer/user direction.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -63,6 +63,26 @@ default Map<String, String> getFunctionConfig() {
     return null;
   }
 
+  /// Returns whether the exposed function configuration is null or empty. This does not imply that other settings,
+  /// such as the partition id normalizer, have their default values.
+  @JsonIgnore
+  default boolean hasEmptyConfig() {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Kept hasEmptyConfig public as requested for function implementations, and documented that it is a reusable helper for canReusePartitionIds overrides. The wording continues to distinguish an empty exposed map from a default normalizer, and JsonIgnore remains.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -94,6 +98,13 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     if (filterExpression == null) {
       return segments;
     }
+    int numSegments = segments.size();
+    if (numSegments == 0) {
+      return segments;
+    }
+    if (numSegments >= MIN_SEGMENTS_FOR_PREPARATION) {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Added the query-level escape hatch enablePartitionPruningCache=false, which uses the original evaluator while retaining partition pruning. Default/true still honors the 256-candidate guard, so this is not a force-on switch for small inputs. Shared predicate, invalid-metadata, refresh, and large-config fixtures now run at 255/256 and with the cache disabled; the call-count test proves the switch restores 256 evaluations rather than one. The separate single-evaluator design discussion remains open above, and the mixed-case benchmark caveat remains in the description.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +116,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    PreparedPredicate predicate = null;
+    PartitionFunction cachedFunction = null;
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      if (predicate == null) {
+        predicate = new PreparedPredicate(filterExpression);
+        cachedFunction = function;
+      }
+      // Reuse the filter structure even when the functions cannot share partition ids.
+      boolean reusePartitionIds = cachedFunction.canReusePartitionIds(function);

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Added the same-instance fast path in PartitionFunction.canReusePartitionIds, so configured self-comparisons are now true. Distinct configured instances still conservatively fall back unless an implementation such as Custom opts in. The single pivot remains intentional: a first-segment outlier can reduce the optimization's benefit, but does not change results. Re-pivoting or a per-function map is deferred to preserve the requested minimal query-local design; that limitation is documented.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +116,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {
+    Set<String> selectedSegments = new HashSet<>();
+    PreparedPredicate predicate = null;
+    PartitionFunction cachedFunction = null;
+    for (String segment : segments) {
+      SegmentPartitionInfo partitionInfo = _partitionInfoMap.get(segment);
+      if (partitionInfo == null || partitionInfo == SegmentPartitionUtils.INVALID_PARTITION_INFO) {
+        selectedSegments.add(segment);
+        continue;
+      }
+      PartitionFunction function = partitionInfo.getPartitionFunction();
+      if (predicate == null) {
+        predicate = new PreparedPredicate(filterExpression);
+        cachedFunction = function;
+      }
+      // Reuse the filter structure even when the functions cannot share partition ids.
+      boolean reusePartitionIds = cachedFunction.canReusePartitionIds(function);
+      if (predicate.matches(partitionInfo.getPartitions(), function, reusePartitionIds)) {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Deferring this extra optimization to keep the change small. _partitionIds == null is not by itself proof of constant true: children may be unvisited due to short-circuiting, and an empty OR returns false despite having no partition leaves. Establishing constant truth for the whole expression needs more state/analysis while preserving lazy exceptions. The unrelated-column regression now exercises both evaluators at the cutoff; the current conservative loop remains.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +187,82 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private List<Integer> _partitionIds;
+
+    private PreparedPredicate(Expression expression) {
+      _expression = expression;
+    }
+
+    private boolean matches(Set<Integer> partitions, PartitionFunction partitionFunction, boolean reusePartitionIds) {
+      if (_kind == null) {
+        Function function = _expression.getFunctionCall();
+        _kind = FilterKind.valueOf(function.getOperator());
+        _operands = function.getOperands();
+        if (_kind == FilterKind.AND || _kind == FilterKind.OR) {
+          _children = new PreparedPredicate[_operands.size()];
+          for (int i = 0; i < _children.length; i++) {
+            _children[i] = new PreparedPredicate(_operands.get(i));
+          }
+        } else if (_kind == FilterKind.EQUALS || _kind == FilterKind.IN) {
+          Identifier identifier = _operands.get(0).getIdentifier();
+          if (identifier != null && identifier.getName().equals(_partitionColumn)) {
+            _partitionIds = new ArrayList<>(1);

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Kept incremental capacity deliberately and added the rationale beside the allocation. The number of visited literals is not known in advance: an IN with 4,096 values may match its first value on every segment. Allocating for all operands would add proportional per-query allocation even in that case. Boxed cached IDs still avoid re-boxing per segment. A different growth policy should be measured across both full-scan and early-match workloads rather than assuming full preallocation is always cheaper.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +187,82 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Lazily prepares only visited expressions and values. Instances belong to one prune call, never shared by queries.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private FilterKind _kind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private List<Integer> _partitionIds;
+
+    private PreparedPredicate(Expression expression) {
+      _expression = expression;
+    }
+
+    private boolean matches(Set<Integer> partitions, PartitionFunction partitionFunction, boolean reusePartitionIds) {
+      if (_kind == null) {
+        Function function = _expression.getFunctionCall();
+        _kind = FilterKind.valueOf(function.getOperator());
+        _operands = function.getOperands();
+        if (_kind == FilterKind.AND || _kind == FilterKind.OR) {
+          _children = new PreparedPredicate[_operands.size()];
+          for (int i = 0; i < _children.length; i++) {
+            _children[i] = new PreparedPredicate(_operands.get(i));
+          }
+        } else if (_kind == FilterKind.EQUALS || _kind == FilterKind.IN) {
+          Identifier identifier = _operands.get(0).getIdentifier();
+          if (identifier != null && identifier.getName().equals(_partitionColumn)) {
+            _partitionIds = new ArrayList<>(1);
+          }
+        }
+      }
+      switch (_kind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions, partitionFunction, reusePartitionIds)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions, partitionFunction, reusePartitionIds)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds != null) {
+            int numValues = _kind == FilterKind.EQUALS ? 1 : _operands.size() - 1;
+            if (!reusePartitionIds) {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Thanks for verifying the visited-prefix invariant. It is preserved, including the incompatible-function fallback and lazy short-circuit behavior.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -94,6 +98,13 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     if (filterExpression == null) {
       return segments;
     }
+    int numSegments = segments.size();
+    if (numSegments == 0) {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Removed the explicit empty-candidate branch. An empty input falls through the original loop, returns a new empty set, and never evaluates the filter. Existing empty-input tests remain.



##########
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java:
##########
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.routing.segmentpruner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.partition.PartitionIdNormalizer;
+import org.apache.pinot.segment.spi.partition.metadata.ColumnPartitionMetadata;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises query-local partition ID caching with real metadata initialization and refresh, without ZooKeeper.
+public class SinglePartitionColumnSegmentPrunerTest {
+  private static final String COLUMN = "memberId";
+  private static final String TABLE = "testTable_OFFLINE";
+
+  @Test
+  public void testHashesOnceAcrossDistinctMetadataInstancesPerQuery() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    Set<String> expected = new HashSet<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(i % 8), i % 2 == 0 ? null : Map.of()));
+      if (i % 8 == 3) {
+        expected.add(segment);
+      }
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 1);
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 2, "Computed hashes must not survive a prune call");
+  }
+
+  @Test
+  public void testInterleavedFunctionConfigurationsAndPartitionCounts() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("a", metadata("a", "PrunerCounting", 8, Set.of(3), null));
+    records.put("sameFunction", metadata("sameFunction", "PrunerCounting", 8, Set.of(2), null));
+    records.put("b", metadata("b", "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    records.put("c", metadata("c", "PrunerCounting", 8, Set.of(2), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("e", metadata("e", "PrunerCounting", 16, Set.of(11), null));
+    Set<String> expected = new HashSet<>(Set.of("a", "b", "e"));
+    for (int i = 0; i < 250; i++) {
+      String segment = "tail_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner.prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 4,
+        "Only compatible default functions reuse IDs; configured functions never compare configuration contents");
+    records.put("b", metadata("b", "PrunerCounting", 16, Set.of(11), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), null));
+    expected.add("d");
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 3,
+        "Different partition counts must not reuse partition IDs");
+  }
+
+  @Test
+  public void testDuplicateInPartitionsAndIncrementalEvaluation() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(1), null));
+    records.put("configured", metadata("configured", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("second", metadata("second", "PrunerCounting", 8, Set.of(2), null));
+    records.put("miss", metadata("miss", "PrunerCounting", 8, Set.of(3, 4), null));
+    records.put("repeat", metadata("repeat", "PrunerCounting", 8, Set.of(1, 2), null));
+    Set<String> expected = new HashSet<>(Set.of("first", "configured", "second", "repeat"));
+    for (int i = 0; i < 252; i++) {
+      String segment = "repeat_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(1, 2), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    // The configured segment must not consume or extend the prefix cached by the first segment.
+    assertEquals(pruner.prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()),
+        expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 8);
+    // A configured first segment must not seed IDs for later default-config segments.
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(2), Map.of("offset", "1")));
+    assertEquals(pruner(records).prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()), expected);
+  }
+
+  @Test
+  public void testLargeConfigurationsAndUnrelatedPredicates() throws Exception {
+    String values = "first|" + "x".repeat(100_000);
+    Map<String, String> config = Map.of("columnValues", values, "columnValuesDelimiter", "|");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), config);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), config);
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("first", first, "second", second));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of("first"));
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("value"))), Set.of("first", "second")), Set.of("first", "second"));
+    pruner.refreshSegment("first", metadata("first", "BoundedColumnValue", 3, Set.of(2), config));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of());
+  }
+
+  @Test
+  public void testConfigurationHashCollisionsDoNotReusePartitionIds() throws Exception {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Removed the obsolete configuration-hash-collision test. The remaining configured-function, mixed-metadata, and expanded large-config fixtures cover the current compatibility/fallback design without referring to the removed key-hashing implementation.



##########
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java:
##########
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.routing.segmentpruner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.partition.PartitionIdNormalizer;
+import org.apache.pinot.segment.spi.partition.metadata.ColumnPartitionMetadata;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises query-local partition ID caching with real metadata initialization and refresh, without ZooKeeper.
+public class SinglePartitionColumnSegmentPrunerTest {
+  private static final String COLUMN = "memberId";
+  private static final String TABLE = "testTable_OFFLINE";
+
+  @Test
+  public void testHashesOnceAcrossDistinctMetadataInstancesPerQuery() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    Set<String> expected = new HashSet<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(i % 8), i % 2 == 0 ? null : Map.of()));
+      if (i % 8 == 3) {
+        expected.add(segment);
+      }
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 1);
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 2, "Computed hashes must not survive a prune call");
+  }
+
+  @Test
+  public void testInterleavedFunctionConfigurationsAndPartitionCounts() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("a", metadata("a", "PrunerCounting", 8, Set.of(3), null));
+    records.put("sameFunction", metadata("sameFunction", "PrunerCounting", 8, Set.of(2), null));
+    records.put("b", metadata("b", "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    records.put("c", metadata("c", "PrunerCounting", 8, Set.of(2), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("e", metadata("e", "PrunerCounting", 16, Set.of(11), null));
+    Set<String> expected = new HashSet<>(Set.of("a", "b", "e"));
+    for (int i = 0; i < 250; i++) {
+      String segment = "tail_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner.prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 4,
+        "Only compatible default functions reuse IDs; configured functions never compare configuration contents");
+    records.put("b", metadata("b", "PrunerCounting", 16, Set.of(11), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), null));
+    expected.add("d");
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 3,
+        "Different partition counts must not reuse partition IDs");
+  }
+
+  @Test
+  public void testDuplicateInPartitionsAndIncrementalEvaluation() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(1), null));
+    records.put("configured", metadata("configured", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("second", metadata("second", "PrunerCounting", 8, Set.of(2), null));
+    records.put("miss", metadata("miss", "PrunerCounting", 8, Set.of(3, 4), null));
+    records.put("repeat", metadata("repeat", "PrunerCounting", 8, Set.of(1, 2), null));
+    Set<String> expected = new HashSet<>(Set.of("first", "configured", "second", "repeat"));
+    for (int i = 0; i < 252; i++) {
+      String segment = "repeat_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(1, 2), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    // The configured segment must not consume or extend the prefix cached by the first segment.
+    assertEquals(pruner.prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()),
+        expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 8);
+    // A configured first segment must not seed IDs for later default-config segments.
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(2), Map.of("offset", "1")));
+    assertEquals(pruner(records).prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()), expected);
+  }
+
+  @Test
+  public void testLargeConfigurationsAndUnrelatedPredicates() throws Exception {
+    String values = "first|" + "x".repeat(100_000);
+    Map<String, String> config = Map.of("columnValues", values, "columnValuesDelimiter", "|");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), config);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), config);
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("first", first, "second", second));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of("first"));
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("value"))), Set.of("first", "second")), Set.of("first", "second"));
+    pruner.refreshSegment("first", metadata("first", "BoundedColumnValue", 3, Set.of(2), config));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of());
+  }
+
+  @Test
+  public void testConfigurationHashCollisionsDoNotReusePartitionIds() throws Exception {
+    Map<String, String> firstConfig = Map.of("columnValues", "Aa|BB", "columnValuesDelimiter", "|");
+    Map<String, String> secondConfig = Map.of("columnValues", "BB|Aa", "columnValuesDelimiter", "|");
+    assertEquals(firstConfig.hashCode(), secondConfig.hashCode(),
+        "Fixture must exercise a configuration hash collision");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), firstConfig);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), secondConfig);
+    Map<String, ZNRecord> records = Map.of("first", first, "second", second);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "Aa")), records.keySet()), records.keySet());
+  }
+
+  @Test
+  public void testMixedFunctionsNormalizersAndFunctionConfig() throws Exception {
+    Map<String, ZNRecord> moduloRecords = new LinkedHashMap<>();
+    moduloRecords.put("positive", metadata("positive", "Modulo", 8, Set.of(7), null));
+    moduloRecords.put("abs", metadata("abs", "Modulo", 8, Set.of(1), Map.of("partitionIdNormalizer", "ABS")));
+    moduloRecords.put("wrongAbs", metadata("wrongAbs", "Modulo", 8, Set.of(7), Map.of("partitionIdNormalizer", "ABS")));
+    Set<String> expected = new HashSet<>(Set.of("positive", "abs"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "positive_" + i;
+      moduloRecords.put(segment, metadata(segment, "Modulo", 8, Set.of(7), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(moduloRecords).prune(request(predicate("EQUALS", "-1")), moduloRecords.keySet()),
+        expected);
+
+    String value = "80ff0102";
+    Map<String, String> rawConfig = Map.of("useRawBytes", "true");
+    int textPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, null).getPartition(value);
+    int rawPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, rawConfig).getPartition(value);
+    assertNotEquals(textPartition, rawPartition, "Fixture must distinguish raw-byte and string hashing");
+    Map<String, ZNRecord> murmurRecords = new LinkedHashMap<>();
+    murmurRecords.put("text", metadata("text", "Murmur", 97, Set.of(textPartition), null));
+    murmurRecords.put("raw", metadata("raw", "Murmur", 97, Set.of(rawPartition), rawConfig));
+    murmurRecords.put("wrongRaw", metadata("wrongRaw", "Murmur", 97, Set.of(textPartition), rawConfig));
+    expected = new HashSet<>(Set.of("text", "raw"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "text_" + i;
+      murmurRecords.put(segment, metadata(segment, "Murmur", 97, Set.of(textPartition), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(murmurRecords).prune(request(predicate("EQUALS", value)), murmurRecords.keySet()),
+        expected);
+
+    Map<String, ZNRecord> lookupRecords = new LinkedHashMap<>();
+    lookupRecords.put("first", metadata("first", "BoundedColumnValue", 3, Set.of(1),
+        Map.of("columnValues", "11|12", "columnValuesDelimiter", "|")));
+    lookupRecords.put("second", metadata("second", "BoundedColumnValue", 3, Set.of(2),
+        Map.of("columnValues", "12|11", "columnValuesDelimiter", "|")));
+    lookupRecords.put("modulo", metadata("modulo", "Modulo", 3, Set.of(2), null));
+    assertEquals(pruner(lookupRecords).prune(request(predicate("EQUALS", "11")), lookupRecords.keySet()),
+        lookupRecords.keySet());
+  }
+
+  @DataProvider
+  public Object[][] candidateCounts() {
+    return new Object[][]{{1}, {2}, {256}};
+  }
+
+  @Test(dataProvider = "candidateCounts")
+  public void testAndOrUnsupportedPredicatesAndLazyInValues(int numSegments) throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < numSegments; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "Modulo", 8, Set.of(1), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    Set<String> segments = records.keySet();
+    Expression invalidValue = predicate("EQUALS", "invalid-number");
+    assertEquals(pruner.prune(request(predicate("IN", "1", "invalid-number")), segments), segments);
+    expectThrows(NumberFormatException.class,
+        () -> pruner.prune(request(predicate("IN", "2", "invalid-number")), segments));
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidValue)), segments), Set.of());
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidValue)), segments), segments);
+    Expression invalidOperator = function("INVALID_OPERATOR");
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidOperator)), segments), segments);
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidOperator)), segments), Set.of());
+    expectThrows(IllegalArgumentException.class,
+        () -> pruner.prune(request(function("OR", predicate("EQUALS", "2"), invalidOperator)), segments));
+    expectThrows(IllegalArgumentException.class, () -> pruner.prune(request(invalidOperator), segments));
+
+    Expression unsupported = predicate("GREATER_THAN", "100");
+    assertEquals(pruner.prune(request(function("AND", predicate("IN", "0", "1"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "2"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("NOT", predicate("EQUALS", "1"))), segments), segments);
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    Expression transformedColumn = function("LOWER", RequestUtils.getIdentifierExpression(COLUMN));
+    assertEquals(pruner.prune(request(function("EQUALS", transformedColumn,
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    BrokerRequest unfilteredRequest = new BrokerRequest();
+    unfilteredRequest.setPinotQuery(new PinotQuery());
+    assertSame(pruner.prune(unfilteredRequest, segments), segments);
+  }
+
+  @Test
+  public void testEmptyCandidatesDoNotEvaluateFilter() throws Exception {
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("one", metadata("one", "Modulo", 8, Set.of(1), null)));
+    assertEquals(pruner.prune(request(function("INVALID_OPERATOR")), Set.of()), Set.of());
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), Set.of()), Set.of());
+  }
+
+  @Test
+  public void testUnknownMetadataIsConservativeAndDoesNotEvaluateFilter() throws Exception {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Expanded both fixtures through the shared provider: 255/256 candidates and cache-disabled 256 candidates, as well as the existing small cases. Unknown metadata still preserves every candidate without decoding invalid operators or values; the large-config fixture now actually reaches the prepared path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4067650268


##########
pinot-broker/src/test/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPrunerTest.java:
##########
@@ -0,0 +1,393 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.broker.routing.segmentpruner;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicInteger;
+import javax.annotation.Nullable;
+import org.apache.helix.zookeeper.datamodel.ZNRecord;
+import org.apache.pinot.common.metadata.segment.SegmentPartitionMetadata;
+import org.apache.pinot.common.request.BrokerRequest;
+import org.apache.pinot.common.request.Expression;
+import org.apache.pinot.common.request.PinotQuery;
+import org.apache.pinot.common.utils.request.RequestUtils;
+import org.apache.pinot.segment.spi.partition.PartitionFunction;
+import org.apache.pinot.segment.spi.partition.PartitionFunctionFactory;
+import org.apache.pinot.segment.spi.partition.PartitionIdNormalizer;
+import org.apache.pinot.segment.spi.partition.metadata.ColumnPartitionMetadata;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.testng.annotations.DataProvider;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertNotEquals;
+import static org.testng.Assert.assertSame;
+import static org.testng.Assert.expectThrows;
+
+
+/// Exercises query-local partition ID caching with real metadata initialization and refresh, without ZooKeeper.
+public class SinglePartitionColumnSegmentPrunerTest {
+  private static final String COLUMN = "memberId";
+  private static final String TABLE = "testTable_OFFLINE";
+
+  @Test
+  public void testHashesOnceAcrossDistinctMetadataInstancesPerQuery() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    Set<String> expected = new HashSet<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(i % 8), i % 2 == 0 ? null : Map.of()));
+      if (i % 8 == 3) {
+        expected.add(segment);
+      }
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 1);
+    assertEquals(pruner.prune(request, records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 2, "Computed hashes must not survive a prune call");
+  }
+
+  @Test
+  public void testInterleavedFunctionConfigurationsAndPartitionCounts() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("a", metadata("a", "PrunerCounting", 8, Set.of(3), null));
+    records.put("sameFunction", metadata("sameFunction", "PrunerCounting", 8, Set.of(2), null));
+    records.put("b", metadata("b", "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    records.put("c", metadata("c", "PrunerCounting", 8, Set.of(2), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("e", metadata("e", "PrunerCounting", 16, Set.of(11), null));
+    Set<String> expected = new HashSet<>(Set.of("a", "b", "e"));
+    for (int i = 0; i < 250; i++) {
+      String segment = "tail_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner.prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 4,
+        "Only compatible default functions reuse IDs; configured functions never compare configuration contents");
+    records.put("b", metadata("b", "PrunerCounting", 16, Set.of(11), null));
+    records.put("d", metadata("d", "PrunerCounting", 8, Set.of(3), null));
+    expected.add("d");
+    CountingPartitionFunction.CALLS.set(0);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "11")), records.keySet()), expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 3,
+        "Different partition counts must not reuse partition IDs");
+  }
+
+  @Test
+  public void testDuplicateInPartitionsAndIncrementalEvaluation() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(1), null));
+    records.put("configured", metadata("configured", "PrunerCounting", 8, Set.of(3), Map.of("offset", "1")));
+    records.put("second", metadata("second", "PrunerCounting", 8, Set.of(2), null));
+    records.put("miss", metadata("miss", "PrunerCounting", 8, Set.of(3, 4), null));
+    records.put("repeat", metadata("repeat", "PrunerCounting", 8, Set.of(1, 2), null));
+    Set<String> expected = new HashSet<>(Set.of("first", "configured", "second", "repeat"));
+    for (int i = 0; i < 252; i++) {
+      String segment = "repeat_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(1, 2), null));
+      expected.add(segment);
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    CountingPartitionFunction.CALLS.set(0);
+    // The configured segment must not consume or extend the prefix cached by the first segment.
+    assertEquals(pruner.prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()),
+        expected);
+    assertEquals(CountingPartitionFunction.CALLS.get(), 8);
+    // A configured first segment must not seed IDs for later default-config segments.
+    records.put("first", metadata("first", "PrunerCounting", 8, Set.of(2), Map.of("offset", "1")));
+    assertEquals(pruner(records).prune(request(predicate("IN", "1", "9", "17", "2")), records.keySet()), expected);
+  }
+
+  @Test
+  public void testLargeConfigurationsAndUnrelatedPredicates() throws Exception {
+    String values = "first|" + "x".repeat(100_000);
+    Map<String, String> config = Map.of("columnValues", values, "columnValuesDelimiter", "|");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), config);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), config);
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("first", first, "second", second));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of("first"));
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("value"))), Set.of("first", "second")), Set.of("first", "second"));
+    pruner.refreshSegment("first", metadata("first", "BoundedColumnValue", 3, Set.of(2), config));
+    assertEquals(pruner.prune(request(predicate("EQUALS", "first")), Set.of("first", "second")), Set.of());
+  }
+
+  @Test
+  public void testConfigurationHashCollisionsDoNotReusePartitionIds() throws Exception {
+    Map<String, String> firstConfig = Map.of("columnValues", "Aa|BB", "columnValuesDelimiter", "|");
+    Map<String, String> secondConfig = Map.of("columnValues", "BB|Aa", "columnValuesDelimiter", "|");
+    assertEquals(firstConfig.hashCode(), secondConfig.hashCode(),
+        "Fixture must exercise a configuration hash collision");
+    ZNRecord first = metadata("first", "BoundedColumnValue", 3, Set.of(1), firstConfig);
+    ZNRecord second = metadata("second", "BoundedColumnValue", 3, Set.of(2), secondConfig);
+    Map<String, ZNRecord> records = Map.of("first", first, "second", second);
+    assertEquals(pruner(records).prune(request(predicate("EQUALS", "Aa")), records.keySet()), records.keySet());
+  }
+
+  @Test
+  public void testMixedFunctionsNormalizersAndFunctionConfig() throws Exception {
+    Map<String, ZNRecord> moduloRecords = new LinkedHashMap<>();
+    moduloRecords.put("positive", metadata("positive", "Modulo", 8, Set.of(7), null));
+    moduloRecords.put("abs", metadata("abs", "Modulo", 8, Set.of(1), Map.of("partitionIdNormalizer", "ABS")));
+    moduloRecords.put("wrongAbs", metadata("wrongAbs", "Modulo", 8, Set.of(7), Map.of("partitionIdNormalizer", "ABS")));
+    Set<String> expected = new HashSet<>(Set.of("positive", "abs"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "positive_" + i;
+      moduloRecords.put(segment, metadata(segment, "Modulo", 8, Set.of(7), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(moduloRecords).prune(request(predicate("EQUALS", "-1")), moduloRecords.keySet()),
+        expected);
+
+    String value = "80ff0102";
+    Map<String, String> rawConfig = Map.of("useRawBytes", "true");
+    int textPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, null).getPartition(value);
+    int rawPartition = PartitionFunctionFactory.getPartitionFunction("Murmur", 97, rawConfig).getPartition(value);
+    assertNotEquals(textPartition, rawPartition, "Fixture must distinguish raw-byte and string hashing");
+    Map<String, ZNRecord> murmurRecords = new LinkedHashMap<>();
+    murmurRecords.put("text", metadata("text", "Murmur", 97, Set.of(textPartition), null));
+    murmurRecords.put("raw", metadata("raw", "Murmur", 97, Set.of(rawPartition), rawConfig));
+    murmurRecords.put("wrongRaw", metadata("wrongRaw", "Murmur", 97, Set.of(textPartition), rawConfig));
+    expected = new HashSet<>(Set.of("text", "raw"));
+    for (int i = 0; i < 253; i++) {
+      String segment = "text_" + i;
+      murmurRecords.put(segment, metadata(segment, "Murmur", 97, Set.of(textPartition), null));
+      expected.add(segment);
+    }
+    assertEquals(pruner(murmurRecords).prune(request(predicate("EQUALS", value)), murmurRecords.keySet()),
+        expected);
+
+    Map<String, ZNRecord> lookupRecords = new LinkedHashMap<>();
+    lookupRecords.put("first", metadata("first", "BoundedColumnValue", 3, Set.of(1),
+        Map.of("columnValues", "11|12", "columnValuesDelimiter", "|")));
+    lookupRecords.put("second", metadata("second", "BoundedColumnValue", 3, Set.of(2),
+        Map.of("columnValues", "12|11", "columnValuesDelimiter", "|")));
+    lookupRecords.put("modulo", metadata("modulo", "Modulo", 3, Set.of(2), null));
+    assertEquals(pruner(lookupRecords).prune(request(predicate("EQUALS", "11")), lookupRecords.keySet()),
+        lookupRecords.keySet());
+  }
+
+  @DataProvider
+  public Object[][] candidateCounts() {
+    return new Object[][]{{1}, {2}, {256}};
+  }
+
+  @Test(dataProvider = "candidateCounts")
+  public void testAndOrUnsupportedPredicatesAndLazyInValues(int numSegments) throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < numSegments; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "Modulo", 8, Set.of(1), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    Set<String> segments = records.keySet();
+    Expression invalidValue = predicate("EQUALS", "invalid-number");
+    assertEquals(pruner.prune(request(predicate("IN", "1", "invalid-number")), segments), segments);
+    expectThrows(NumberFormatException.class,
+        () -> pruner.prune(request(predicate("IN", "2", "invalid-number")), segments));
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidValue)), segments), Set.of());
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidValue)), segments), segments);
+    Expression invalidOperator = function("INVALID_OPERATOR");
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "1"), invalidOperator)), segments), segments);
+    assertEquals(pruner.prune(request(function("AND", predicate("EQUALS", "2"), invalidOperator)), segments), Set.of());
+    expectThrows(IllegalArgumentException.class,
+        () -> pruner.prune(request(function("OR", predicate("EQUALS", "2"), invalidOperator)), segments));
+    expectThrows(IllegalArgumentException.class, () -> pruner.prune(request(invalidOperator), segments));
+
+    Expression unsupported = predicate("GREATER_THAN", "100");
+    assertEquals(pruner.prune(request(function("AND", predicate("IN", "0", "1"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("OR", predicate("EQUALS", "2"), unsupported)), segments), segments);
+    assertEquals(pruner.prune(request(function("NOT", predicate("EQUALS", "1"))), segments), segments);
+    assertEquals(pruner.prune(request(function("EQUALS", RequestUtils.getIdentifierExpression("other"),
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    Expression transformedColumn = function("LOWER", RequestUtils.getIdentifierExpression(COLUMN));
+    assertEquals(pruner.prune(request(function("EQUALS", transformedColumn,
+        RequestUtils.getLiteralExpression("invalid-number"))), segments), segments);
+    BrokerRequest unfilteredRequest = new BrokerRequest();
+    unfilteredRequest.setPinotQuery(new PinotQuery());
+    assertSame(pruner.prune(unfilteredRequest, segments), segments);
+  }
+
+  @Test
+  public void testEmptyCandidatesDoNotEvaluateFilter() throws Exception {
+    SinglePartitionColumnSegmentPruner pruner = pruner(Map.of("one", metadata("one", "Modulo", 8, Set.of(1), null)));
+    assertEquals(pruner.prune(request(function("INVALID_OPERATOR")), Set.of()), Set.of());
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), Set.of()), Set.of());
+  }
+
+  @Test
+  public void testUnknownMetadataIsConservativeAndDoesNotEvaluateFilter() throws Exception {
+    SinglePartitionColumnSegmentPruner pruner = new SinglePartitionColumnSegmentPruner(TABLE, COLUMN);
+    ZNRecord invalid = new ZNRecord("invalid");
+    invalid.setSimpleField(CommonConstants.Segment.PARTITION_METADATA, "invalid-json");
+    pruner.init(null, null, List.of("missing", "empty", "invalid"),
+        Arrays.asList(null, new ZNRecord("empty"), invalid));
+    Set<String> segments = Set.of("missing", "empty", "invalid", "not-initialized");
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), segments), segments);
+    assertEquals(pruner.prune(request(function("INVALID_OPERATOR")), segments), segments);
+  }
+
+  @Test(dataProvider = "candidateCounts")
+  public void testRefreshUsesCurrentPartitionsAndConfiguration(int numSegments) throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < numSegments; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "PrunerCounting", 8, Set.of(3), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    Set<String> segments = records.keySet();
+    BrokerRequest request = request(predicate("EQUALS", "3"));
+    assertEquals(pruner.prune(request, segments), segments);
+    for (String segment : segments) {
+      pruner.refreshSegment(segment, metadata(segment, "PrunerCounting", 8, Set.of(4), null));
+    }
+    assertEquals(pruner.prune(request, segments), Set.of());
+    for (String segment : segments) {
+      pruner.refreshSegment(segment, metadata(segment, "PrunerCounting", 8, Set.of(4), Map.of("offset", "1")));
+    }
+    assertEquals(pruner.prune(request, segments), segments);
+    for (String segment : segments) {
+      pruner.refreshSegment(segment, null);
+    }
+    assertEquals(pruner.prune(request(predicate("EQUALS", "invalid-number")), segments), segments);
+  }
+
+  @Test
+  public void testConcurrentQueriesKeepSeparateCachedValues() throws Exception {
+    Map<String, ZNRecord> records = new LinkedHashMap<>();
+    for (int i = 0; i < 256; i++) {
+      String segment = "segment_" + i;
+      records.put(segment, metadata(segment, "Modulo", 8, Set.of(i % 8), null));
+    }
+    SinglePartitionColumnSegmentPruner pruner = pruner(records);
+    ExecutorService executor = Executors.newFixedThreadPool(4);
+    try {
+      List<Callable<Void>> queries = new ArrayList<>();
+      for (int i = 0; i < 32; i++) {
+        int partition = i % 8;
+        Set<String> expected = new HashSet<>();
+        for (int segment = partition; segment < 256; segment += 8) {
+          expected.add("segment_" + segment);
+        }
+        queries.add(() -> {
+          assertEquals(pruner.prune(request(predicate("EQUALS", Integer.toString(partition))), records.keySet()),
+              expected);
+          return null;
+        });
+      }
+      for (Future<Void> future : executor.invokeAll(queries)) {
+        future.get();
+      }
+    } finally {
+      executor.shutdownNow();
+    }
+  }
+
+  private static SinglePartitionColumnSegmentPruner pruner(Map<String, ZNRecord> records) {
+    SinglePartitionColumnSegmentPruner pruner = new SinglePartitionColumnSegmentPruner(TABLE, COLUMN);
+    pruner.init(null, null, new ArrayList<>(records.keySet()), new ArrayList<>(records.values()));
+    return pruner;
+  }
+
+  private static ZNRecord metadata(String segment, String function, int count, Set<Integer> partitions,
+      @Nullable Map<String, String> config) throws Exception {
+    ZNRecord record = new ZNRecord(segment);
+    record.setSimpleField(CommonConstants.Segment.PARTITION_METADATA, new SegmentPartitionMetadata(Map.of(COLUMN,
+        new ColumnPartitionMetadata(function, count, partitions, config))).toJsonString());
+    return record;
+  }
+
+  private static Expression predicate(String operator, String... values) {
+    List<Expression> operands = new ArrayList<>();
+    operands.add(RequestUtils.getIdentifierExpression(COLUMN));
+    for (String value : values) {
+      operands.add(RequestUtils.getLiteralExpression(value));
+    }
+    return RequestUtils.getFunctionExpression(operator, operands);
+  }
+
+  private static Expression function(String operator, Expression... operands) {
+    return RequestUtils.getFunctionExpression(operator, List.of(operands));
+  }
+
+  private static BrokerRequest request(Expression filter) {
+    PinotQuery pinotQuery = new PinotQuery();
+    pinotQuery.setFilterExpression(filter);
+    BrokerRequest brokerRequest = new BrokerRequest();
+    brokerRequest.setPinotQuery(pinotQuery);
+    return brokerRequest;
+  }
+
+  /// Stateless partition function with observable hash calls.
+  public static class CountingPartitionFunction implements PartitionFunction {
+    private static final long serialVersionUID = 1L;
+    private static final AtomicInteger CALLS = new AtomicInteger();

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Documented the sequential-method requirement and factory registration on CountingPartitionFunction. The concurrent-query fixture deliberately uses Modulo rather than the shared counter, so its parallel calls do not interfere with exact-count assertions.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -105,6 +116,30 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
     return selectedSegments;
   }
 
+  private Set<String> pruneWithPreparedPredicate(Expression filterExpression, Set<String> segments) {

Review Comment:
   Follow-up in d9135a1518 (38 focused Apache cases passed): Documented explicitly that this PR only optimizes SinglePartitionColumnSegmentPruner. MultiPartitionColumnsSegmentPruner remains unchanged and is follow-up scope; this does not claim table-wide coverage for multi-column partitioning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4070957576


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   Revisited against current head 115e893c3f. Yes, preparing query-local state can cost more than direct evaluation for small or non-reusable workloads. The cutoff comparison at eight segments showed the configured fallback IN control at 1.4352 -> 1.4927 us (+4.0%); at 32 it was 6.8524 -> 6.7020 us. This is why the current PR retains the allocation-free legacy evaluator below the measured default of 32 candidates, with live global/per-table overrides and enablePartitionPruningCache=false as an escape hatch. Those measurements are workload-specific, not a universal break-even claim.
   
   For this PR I am retaining the measured two-path design. The benefit of a single evaluator is real, but unconditionally preparing it would change the small-query path; a shared nullable-cache evaluator would also need its own performance comparison. The current boundary tests run the same predicate fixtures at 31/32 candidates and with the cache forced off, including lazy exceptions and unknown metadata. Zero candidates already return a fresh empty set without evaluating the filter or preparing a predicate.
   
   Resolving with this explicit performance/maintenance tradeoff, not claiming the evaluators have been consolidated. Consolidation can be revisited separately with equivalent small-query and fallback benchmarks. Current head retains the previously verified 78 focused Apache and 46 StarTree passing cases; no source changes were made during this re-review.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4041153763


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   Non-blocking performance concern: this compares the full configuration map for every subsequent segment, even when the filter only references an unrelated column. Segment metadata is deserialized independently, so for `BoundedColumnValue` the comparison scans the entire `columnValues` string. With 1,000 segments and a 100 KB configuration string, that adds roughly 100 MB of text comparisons per prune call; the previous unrelated-column path returned after checking the identifier. Equality against the first configured bounded value has the same issue. Could we canonicalize equivalent configurations to shared immutable identities during metadata initialization/refresh, retaining full equality there, and compare identities in the query loop? The reported Murmur benchmarks do not cover this configuration-dependent cost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Reuse partition predicate hashes across broker segments [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4042063613


##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {

Review Comment:
   Agreed. I kept the existing isPartitionMatch() method unchanged; the previous diff view was showing the old side of the change.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {

Review Comment:
   The wrapper keeps two key roles separate: CachedPartitionFunctionKey owns an immutable key retained in the map, while PartitionFunctionLookup is a mutable probe reused for lookups and is never inserted. Putting equality on SegmentPartitionInfo would make its segment-specific partition set part of an otherwise unrelated identity, so I kept the small key wrapper.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -95,10 +98,12 @@ public Set<String> prune(BrokerRequest brokerRequest, Set<String> segments) {
       return segments;
     }
     Set<String> selectedSegments = new HashSet<>();
+    // A singleton has no repeated work to reuse. Keep its evaluation free of predicate/cache setup.
+    QueryPartitionMatcher matcher = segments.size() > 1 ? new QueryPartitionMatcher(filterExpression) : null;

Review Comment:
   I kept the singleton fast path and only create QueryPartitionMatcher when there is more than one segment. The matcher removes repeated predicate parsing and partition hashing while preserving the original isPartitionMatch() path for the no-reuse case.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {

Review Comment:
   Addressed in 5a1bbd7585: PreparedPredicate now prepares in its constructor and its fields are final.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;

Review Comment:
   I kept the lazily populated Integer[] because it preserves AND/OR short-circuit behavior and avoids hashing later IN values when an earlier value already matches. Duplicate partition IDs are harmless for membership checks; deduplicating would require eagerly evaluating all values or adding another per-predicate structure.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -126,30 +131,180 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return false;
       case EQUALS: {
         Identifier identifier = operands.get(0).getIdentifier();
-        if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          return partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-              .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
-        } else {
-          return true;
-        }
+        return identifier == null || !identifier.getName().equals(_partitionColumn)
+            || partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+                .getPartition(RequestContextUtils.getStringValue(operands.get(1))));
       }
       case IN: {
         Identifier identifier = operands.get(0).getIdentifier();
+        if (identifier == null || !identifier.getName().equals(_partitionColumn)) {
+          return true;
+        }
+        for (int i = 1; i < operands.size(); i++) {
+          if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
+              .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+            return true;
+          }
+        }
+        return false;
+      }
+      default:
+        return true;
+    }
+  }
+
+  /// All prepared predicates and hashes belong to one prune call; refreshes and other queries share none of this state.
+  private final class QueryPartitionMatcher {
+    private final Expression _filterExpression;
+    private final Map<PartitionFunctionKey, PreparedPredicate> _predicates = new HashMap<>();
+    private final PartitionFunctionLookup _lookup = new PartitionFunctionLookup();
+    private PreparedPredicate _lastPredicate;
+
+    private QueryPartitionMatcher(Expression filterExpression) {
+      _filterExpression = filterExpression;
+    }
+
+    private boolean matches(SegmentPartitionInfo partitionInfo) {
+      // Segment metadata contains distinct function instances. Avoid allocating a key per segment for the common
+      // case where those instances have identical configuration.
+      if (_lastPredicate == null || !_lookup.matches(partitionInfo)) {
+        // This reusable lookup probe is never inserted. Only a previously unseen configuration allocates a stored key.
+        _lookup._partitionInfo = partitionInfo;
+        _lastPredicate = _predicates.get(_lookup);
+        if (_lastPredicate == null) {
+          _lastPredicate = new PreparedPredicate(_filterExpression, partitionInfo.getPartitionFunction());
+          _predicates.put(new CachedPartitionFunctionKey(partitionInfo), _lastPredicate);
+        }
+      }
+      return _lastPredicate.matches(partitionInfo.getPartitions());
+    }
+  }
+
+  /// Interprets each visited predicate once and hashes IN values only as far as short-circuit evaluation requires.
+  private final class PreparedPredicate {
+    private final Expression _expression;
+    private final PartitionFunction _partitionFunction;
+    private FilterKind _filterKind;
+    private List<Expression> _operands;
+    private PreparedPredicate[] _children;
+    private Integer[] _partitionIds;
+
+    private PreparedPredicate(Expression expression, PartitionFunction partitionFunction) {
+      _expression = expression;
+      _partitionFunction = partitionFunction;
+    }
+
+    private void prepare() {
+      Function function = _expression.getFunctionCall();
+      _filterKind = FilterKind.valueOf(function.getOperator());
+      _operands = function.getOperands();
+      if (_filterKind == FilterKind.AND || _filterKind == FilterKind.OR) {
+        _children = new PreparedPredicate[_operands.size()];
+        for (int i = 0; i < _children.length; i++) {
+          _children[i] = new PreparedPredicate(_operands.get(i), _partitionFunction);
+        }
+      } else if (_filterKind == FilterKind.EQUALS || _filterKind == FilterKind.IN) {
+        Identifier identifier = _operands.get(0).getIdentifier();
         if (identifier != null && identifier.getName().equals(_partitionColumn)) {
-          int numOperands = operands.size();
-          for (int i = 1; i < numOperands; i++) {
-            if (partitionInfo.getPartitions().contains(partitionInfo.getPartitionFunction()
-                .getPartition(RequestContextUtils.getStringValue(operands.get(i))))) {
+          _partitionIds = new Integer[_filterKind == FilterKind.EQUALS ? 1 : _operands.size() - 1];
+        }
+      }
+    }
+
+    private boolean matches(Set<Integer> partitions) {
+      if (_filterKind == null) {
+        prepare();
+      }
+      switch (_filterKind) {
+        case AND:
+          for (PreparedPredicate child : _children) {
+            if (!child.matches(partitions)) {
+              return false;
+            }
+          }
+          return true;
+        case OR:
+          for (PreparedPredicate child : _children) {
+            if (child.matches(partitions)) {
+              return true;
+            }
+          }
+          return false;
+        case EQUALS:
+        case IN:
+          if (_partitionIds == null) {
+            return true;
+          }
+          for (int i = 0; i < _partitionIds.length; i++) {
+            Integer partitionId = _partitionIds[i];
+            if (partitionId == null) {
+              partitionId = _partitionFunction.getPartition(RequestContextUtils.getStringValue(_operands.get(i + 1)));
+              _partitionIds[i] = partitionId;
+            }
+            if (partitions.contains(partitionId)) {
               return true;
             }
           }
           return false;
-        } else {
+        default:
           return true;
-        }
       }
-      default:
-        return true;
+    }
+  }
+
+  /// Uses all recorded constructor inputs to identify equivalent partition functions.
+  private abstract static class PartitionFunctionKey {
+    abstract SegmentPartitionInfo getPartitionInfo();
+
+    final boolean matches(SegmentPartitionInfo partitionInfo) {
+      SegmentPartitionInfo current = getPartitionInfo();
+      PartitionFunction function = current.getPartitionFunction();
+      PartitionFunction otherFunction = partitionInfo.getPartitionFunction();
+      return function.getClass() == otherFunction.getClass() && function.getName().equals(otherFunction.getName())
+          && function.getNumPartitions() == otherFunction.getNumPartitions()
+          && function.getPartitionIdNormalizer() == otherFunction.getPartitionIdNormalizer()
+          && Objects.equals(current.getPartitionFunctionConfig(), partitionInfo.getPartitionFunctionConfig());

Review Comment:
   Addressed in 5a1bbd7585: SegmentPartitionInfo now computes the configuration hash once during metadata initialization, and the query-loop key comparison checks that primitive hash before falling back to full map equality. This avoids scanning large configuration maps for the normal non-equal case while retaining collision-safe equality.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Prepare partition pruning predicates once per query [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #19539:
URL: https://github.com/apache/pinot/pull/19539#discussion_r4201818348


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/PartitionFunction.java:
##########
@@ -58,6 +62,13 @@ default List<String> getNames() {
   /// @return Number of possible partitions.
   int getNumPartitions();
 
+  /// Whether query-local predicate preparation is useful for this function. Functions whose instances cannot be
+  /// compared cheaply across segments can return false to use the original per-segment pruning path.
+  default boolean supportsPartitionIdPreparation() {
+    return true;

Review Comment:
   Can we default this to `false` and explicitly enable preparation for implementations with suitable value equality? Existing plugins that inherit `Object.equals` automatically take the prepared path, but `SegmentPartitionUtils`/`PartitionFunctionFactory` construct a distinct function instance per segment, so their prepared predicates are never reused. For S >= 32 segments and an IN list of V values whose first value matches every segment, the legacy path performs S partition calculations, while this path performs S * V and retains S candidate sets until the prune call finishes. Making the capability opt-in avoids this CPU/allocation regression for existing plugins.



##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/segmentpruner/SinglePartitionColumnSegmentPruner.java:
##########
@@ -152,4 +237,107 @@ private boolean isPartitionMatch(Expression filterExpression, SegmentPartitionIn
         return true;
     }
   }
+
+  /// Computes conservative candidate IDs for the whole predicate. Null represents every partition.
+  /// Partition columns are single-valued, so a row satisfying AND must belong to every child's candidate set.
+  @Nullable
+  private IntSet preparePartitionIds(Expression expression, PartitionFunction partitionFunction) {
+    Function function = expression.getFunctionCall();
+    FilterKind kind = FilterKind.valueOf(function.getOperator());
+    List<Expression> operands = function.getOperands();
+    switch (kind) {
+      case AND: {
+        IntSet ids = null;
+        for (Expression child : operands) {
+          IntSet childIds = preparePartitionIds(child, partitionFunction);
+          if (childIds != null) {
+            if (ids == null) {
+              ids = childIds;
+            } else {
+              // Singleton leaves are immutable; composite and IN sets can be intersected in place.
+              if (!(ids instanceof IntOpenHashSet)) {
+                ids = new IntOpenHashSet(ids);
+              }
+              ids.retainAll(childIds);
+            }
+          }
+        }
+        return ids;
+      }
+      case OR: {
+        IntSet ids = new IntOpenHashSet();
+        boolean matchesAll = false;
+        for (Expression child : operands) {
+          IntSet childIds = preparePartitionIds(child, partitionFunction);
+          if (childIds == null) {
+            matchesAll = true;

Review Comment:
   Can we return `null` immediately when `childIds == null`? For `otherColumn = 'x' OR memberId IN (<large list>)`, the legacy evaluator returns on the first child without computing any partition IDs. Here we set `matchesAll` but still hash the entire IN list and build a set that is ultimately discarded. An unrestricted child already determines the OR's candidate set, and later siblings were unreachable in the legacy evaluator, so stopping here preserves its short-circuit/error behavior. This also removes the redundant `matchesAll` flag. A counting-function test with an unrelated first child and a large IN sibling would cover it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org