You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@phoenix.apache.org by "kadirozde (via GitHub)" <gi...@apache.org> on 2026/04/28 04:09:36 UTC
[PR] PHOENIX-6791 Replace WHERE optimizer with N-dimensional key-space model [phoenix]
kadirozde opened a new pull request, #2428:
URL: https://github.com/apache/phoenix/pull/2428
Introduces a v2 WHERE optimizer in compile/keyspace/ that models each expression as a list of per-PK-column KeyRange tuples (KeySpace), with AND as per-dim intersection and OR governed by containment + equal-on-N-1 merge rules. Key-range conversion is deferred until trailing dims are dropped against a cartesian bound, making the algorithm O(N^2) in PK arity. RVC inequalities and IN lists are lowered to lexicographic OR-of-AND form in a single ExpressionNormalizer pass so per-dim intersection composes uniformly with every other predicate.
Feature flag phoenix.where.optimizer.v2.enabled routes WhereOptimizer .pushKeyExpressionsToScan to the v2 driver; default is now true. The legacy implementation is untouched and remains the fallback when the flag is disabled. All 138 WhereOptimizerTest methods pass under both flag values; 37 byte-shape divergences are captured in isV2Optimizer() branches with verbose comments explaining why v2 differs from v1 (DESC encoding, RVC clip, tautology simplification, trim-trailing, scalar-function composition limits, filter shape). 37 new keyspace- level unit tests cover the algebra, normalization, visitor, and extractor in isolation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5515924436
After I posted my review I see Copilot added additional comments that may need attention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-6067073147
@kadirozde I updated the merge commit where it failed to address some explain plan changes. Now it does and all relevant tests pass.
While performing the work Opus 5.5 claimed to discover two bugs. They seem legitimate to me so I have included fixes for them on the branch and would like your review if you have time:
- [3e1caf6](https://github.com/apache/phoenix/pull/2428/commits/3e1caf67559ff5d1a07e5bc6b305d69b55fa4729) : When an `IN` list on a leading primary key dimension exceeds the KeySpaceList budget (65,536 spaces), `widenToBudget` completely drops that dimension, turning the scan into an unbounded full table scan.
- [05fed39](https://github.com/apache/phoenix/pull/2428/commits/05fed398df1d87069204e184aeba73e3aced9750): The V2 where optimizer can produce incorrect scan ranges and drop matching rows when filtering on `DESC` variable-length columns (e.g., `VARCHAR DESC`) with prefix-related bounds (such as `k1 = '2' OR k1 > '23'`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192336330
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/ExpressionNormalizer.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.AndExpression;
+import org.apache.phoenix.expression.ComparisonExpression;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.InListExpression;
+import org.apache.phoenix.expression.OrExpression;
+import org.apache.phoenix.expression.RowValueConstructorExpression;
+
+/**
+ * Rewrites a WHERE {@link Expression} into the canonical AND/OR form the v2 key-space model
+ * operates on. Two rewrites are applied bottom-up:
+ * <ul>
+ * <li><b>RVC inequality</b>: {@code (c1,...,cK) OP (v1,...,vK)} for OP in
+ * {@code <, ≤, >, ≥} expands to the lexicographic OR of ANDs. Example:
+ * {@code (c1,c2,c3) > (v1,v2,v3)} becomes
+ * {@code (c1>v1) OR (c1=v1 AND c2>v2) OR (c1=v1 AND c2=v2 AND c3>v3)}. Equality is already
+ * expanded upstream by {@link ComparisonExpression#create}.</li>
+ * <li><b>IN list on a scalar</b>: {@code a IN (v1,...,vK)} expands to
+ * {@code a=v1 OR ... OR a=vK}. IN with an RVC LHS is left intact; the visitor handles
+ * that shape directly by producing one KeySpace per row value.</li>
+ * </ul>
Review Comment:
Corrected the java doc in 9a50c3419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-6084498118
Thank you for the review @kadirozde . I will be back soon with the recommended changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5518229762
I run just the unit test for the last change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5304080420
@apurtell, I updated the PR to resolve merge conflicts. Yes, we can merge it if the reviewers approve the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-4382783298
We will also have to wait until jenkins build machines are up again. For instance, the build is currently failing, but this should be straightforward to fix:
```
[ERROR] Unused declared dependencies found:
[ERROR] org.openjdk.jmh:jmh-generator-annprocess:jar:1.37:test
...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 28.817 s
[INFO] Finished at: 2026-05-05T13:19:03-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.6.1:analyze-only (enforce-dependencies) on project phoenix-core: Dependency problems found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :phoenix-core
```
This is just the reminder that we will have to unblock the build issue, which is separate concern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-6100120618
@kadirozde I addressed your review comments on 05fed39. While performing the work I enabled a workflow that took the feedback of adversarial reviewers into consideration in a loop. This process took 29 findings developed over 9 rounds of review as input and clustered them into 4 root causes, producing 4 additional commits, which were then themselves subjected to a round of adversarial review. The workflow also classified whether a case an adversarial review claimed could be encountered in any realistic query shapes. There were additional findings considered rare or unlikely that were not addressed, in order for this process to converge.
Additional test cases have been added to exercise the cases found during adversarial review. Those new test cases and all existing test cases all pass. As such these four commits ship a converged hardening of the integration of the box algebra on top of the existing complexity of Phoenix's key structure (where it accumulated this complexity over time for supporting ascending and descending sort orders, variable length fields, and nullable fields).
I hope this isn't too much change on top of what has already been delivered here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192325071
##########
phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/algebra/Reference.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.oracle;
+
+/**
+ * Reference implementation (oracle) of the key-space model's key-range extraction
+ * algorithm. Given an {@link AbstractExpression} tree over a schema with {@code nPk}
+ * primary-key dimensions, produces the {@link AbstractKeySpaceList} the algorithm
+ * should emit.
+ * <p>
+ * The purpose is differential testing: we compare the oracle's output against the
+ * production {@code WhereOptimizerV2} implementation's {@code KeySpaceList} to detect
+ * divergences. Any difference is either a production bug or an oracle bug — the oracle
+ * being shorter and directly derived from the design, the default suspect is production.
Review Comment:
This was fixed in a later commit — f5ea31db4 renamed the oracle subpackage to algebra and moved all 5 files (AbstractExpression, AbstractKeySpace, AbstractKeySpaceList, AbstractRange, Reference) from phoenix-core-client/src/main/java/.../keyspace/oracle/ to phoenix-core/src/test/java/.../keyspace/algebra/. The reference-model code no longer ships in the client jar, and the phoenix-core-client/src/main/java/.../keyspace/ tree now only contains the 7 production classes plus the scan/ subpackage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5841086486
First merge attempt (cherry pick of previous merge) did not work, so now I am redoing the merge directly on kadirozde/6791 branch. That will go smoothly. Stand by.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5302223866
I would like us to consider merging this. I have an enhancement program in planning for the optimizer and this represents a significant change risk just hanging out. As it is there will be an interesting delta needed on my EXPLAIN PLAN improvements PR should this get merged in first.
I would argue this should go in just after that, so I can fix up this PR for integration post merge. If you are ok with that approach @kadirozde .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-6084313578
@apurtell Thanks for fixing up the merge commit and for digging into these. I went through both commits.
3e1caf6: bounding hull instead of a full scan when widening a leading dim
This looks good to me. Widening the only remaining constrained dim to EVERYTHING was clearly wrong, and collapsing to the [min, max] hull (the same thing V1 does) with the result marked approximated keeps the IN list as a residual filter, so results stay exact.
- The prefix handling in boundingHull looks right to me. Moving the lower bound to previousKey(min) and the upper bound to nextKey(shortest prefix of max) is correct for both the 0x00 and 0xFF separators.
- Removing the "no progress" bail-out is fine: each pass makes the trailing dim EVERYTHING in every space, so the next trailing dim is strictly lower and the loop always ends.
- Minor: testJoinKeyRangeWithLargeBuildSideIsBounded loads about 73k rows in an IT. That's acceptable, but it adds noticeable runtime.
+1 on this one.
05fed39: DESC variable-length columns with prefix-related bounds
The analysis is right: for a DESC var-length column the row key stores value || 0xFF, so '2' sorts after '23', and every step that compares raw per-column bytes can get the order wrong. A few concerns, though:
1. Possible extra rows when the DESC column gets its own slot. When the compound window is shortened so the trailing DESC column is emitted separately, that slot is now filled with dimRanges(list, d), the union of that column's ranges across all spaces. That loses the pairing with the leading columns. For example, (k1, k2) IN (('a','1'), ('b','10')) with k2 VARCHAR DESC becomes {a,b} x {1,10}, which also admits ('a','10') and ('b','1'). As far as I can see, compoundsApproximated isn't set on this path, and WhereOptimizerV2 drops the visitor-consumed nodes from the residual filter unless r.approximated || keySpaceList.isApproximated(). If the RVC was consumed, those extra rows would come back. WhereOptimizerRowKeyCoverageTest only checks that the scan admits every matching row, so it wouldn't catch this. Could we mark the result approximated whenever the dimRanges branch is taken (or whenever the window is shortened), and add an exact-result test for that RVC IN case against real r
ows?
2. Always stripping the trailing separator. The previous code deliberately kept the separator when unconstrained PK columns follow the compound, and its comment cited QueryCompilerTest.testRVCScanBoundaries1 breaking otherwise. The new code always strips it and relies on ScanUtil.setKey to add it back. Could you confirm testRVCScanBoundaries1 and the other RVC scan-boundary tests still pass with this change?
3. The prefix guards also apply to ASC and fixed-width columns. KeySpace.unionIfMergeable and coalesceColumn have no sort-order information, so they also block merging and coalescing for ASC var-length columns, where byte order already matches row-key order. The prefix check in unionIfMergeable also runs before contains(), so something like k1 LIKE '1%' OR k1 = '10' now keeps two spaces instead of absorbing one. The cost is more spaces, which means hitting the 65,536 bound (and the hull) sooner and producing longer skip-scan slot lists. Would it be feasible to pass in the sort order or schema so the guard only applies to DESC var-length columns?
4. Overlapping ranges in a slot. With prefix bounds present, coalesceColumn now only deduplicates, so a slot can contain overlapping ranges. The comment says SkipScanFilter navigates overlapping ranges. Since it binary-searches its slot ranges, I'd like a targeted test (or a pointer to where that's handled) rather than relying only on the coverage test's examples.
Summary: I'm happy with 3e1caf6 as is. For 05fed39, I think (1) needs a fix before merging, (2) needs confirmation, and (3) and (4) are worth addressing but are less urgent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192318310
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/KeySpace.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace;
+
+import java.util.Arrays;
+import java.util.Optional;
+
+import org.apache.phoenix.query.KeyRange;
+
+/**
+ * An N-dimensional key space over a table's primary key columns. Each dimension is a
+ * {@link KeyRange} over the encoded byte representation of a single PK column. An
+ * expression node is modeled as a list of {@code KeySpace} instances; see
+ * {@link KeySpaceList} for the list-level algebra.
+ * <p>
+ * Instances are immutable. {@link #and(KeySpace)} is the per-dimension intersection;
+ * {@link #unionIfMergeable(KeySpace)} returns the union when either (a) one space contains
+ * the other or (b) the two spaces agree on all but one dimension and the differing dim's
+ * ranges are non-disjoint.
+ * <p>
+ * A single-value predicate like {@code PK2 >= 3} on a 3-PK table is represented as
+ * {@code [(*,*), [3,*), (*,*)]} — a singleton {@link KeySpaceList} containing a single
+ * {@code KeySpace} where every dim not mentioned in the predicate holds
+ * {@link KeyRange#EVERYTHING_RANGE}. RVC inequalities are pre-normalized by
+ * {@link ExpressionNormalizer} into lexicographic AND/OR of scalar comparisons so that
+ * per-dim intersection composes correctly with every other predicate; this class therefore
+ * never needs to model compound-byte concatenation directly.
+ */
+public final class KeySpace {
+
+ private final KeyRange[] dims;
+ private final boolean empty;
+
+ private KeySpace(KeyRange[] dims, boolean empty) {
+ this.dims = dims;
+ this.empty = empty;
+ }
+
+ public static KeySpace everything(int n) {
+ KeyRange[] dims = new KeyRange[n];
+ Arrays.fill(dims, KeyRange.EVERYTHING_RANGE);
+ return new KeySpace(dims, false);
+ }
+
+ public static KeySpace empty(int n) {
+ KeyRange[] dims = new KeyRange[n];
+ Arrays.fill(dims, KeyRange.EMPTY_RANGE);
+ return new KeySpace(dims, true);
+ }
+
+ public static KeySpace single(int dim, KeyRange r, int n) {
+ if (r == KeyRange.EMPTY_RANGE) {
+ return empty(n);
+ }
+ KeyRange[] dims = new KeyRange[n];
+ Arrays.fill(dims, KeyRange.EVERYTHING_RANGE);
+ dims[dim] = r;
+ return new KeySpace(dims, false);
+ }
+
+ public static KeySpace of(KeyRange[] dims) {
+ boolean empty = false;
+ for (KeyRange r : dims) {
+ if (r == KeyRange.EMPTY_RANGE) {
+ empty = true;
+ break;
+ }
+ }
+ return new KeySpace(dims.clone(), empty);
+ }
+
+ public int nDims() {
+ return dims.length;
+ }
+
+ public KeyRange get(int dim) {
+ return dims[dim];
+ }
+
+ /**
+ * Returns a new {@link KeySpace} identical to this one except with dim {@code dim}
+ * replaced by {@code r}. Used by {@link KeySpaceList}'s widening path to drop a
+ * trailing dim (by replacing it with {@link KeyRange#EVERYTHING_RANGE}). Allocates a
+ * fresh dims array; original is unchanged.
+ */
+ public KeySpace withDimReplaced(int dim, KeyRange r) {
+ if (dims[dim].equals(r)) {
+ return this;
+ }
+ KeyRange[] newDims = dims.clone();
+ newDims[dim] = r;
+ if (r == KeyRange.EMPTY_RANGE) {
+ return empty(dims.length);
+ }
+ return new KeySpace(newDims, false);
+ }
+
+ public boolean isEmpty() {
+ return empty;
+ }
+
+ public boolean isEverything() {
+ if (empty) {
+ return false;
+ }
+ for (KeyRange r : dims) {
+ if (r != KeyRange.EVERYTHING_RANGE) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Per-dimension intersection. If any dim collapses to {@link KeyRange#EMPTY_RANGE}, the
+ * result is {@link #empty(int)}.
+ */
+ public KeySpace and(KeySpace other) {
+ requireSameArity(other);
+ if (this.empty || other.empty) {
+ return empty(dims.length);
+ }
+ KeyRange[] newDims = new KeyRange[dims.length];
+ for (int i = 0; i < dims.length; i++) {
+ KeyRange a = this.dims[i];
+ KeyRange b = other.dims[i];
+ KeyRange inter = intersectRange(a, b);
+ if (inter == KeyRange.EMPTY_RANGE) {
+ return empty(dims.length);
+ }
+ newDims[i] = inter;
+ }
+ return new KeySpace(newDims, false);
+ }
+
+ /**
+ * Per-dim intersection that special-cases {@link KeyRange#EVERYTHING_RANGE} against
+ * {@link KeyRange#IS_NULL_RANGE} / {@link KeyRange#IS_NOT_NULL_RANGE}. Plain
+ * {@link KeyRange#intersect} treats EVERYTHING ∩ IS_NULL as EMPTY because IS_NULL uses
+ * an empty-byte-array sentinel that coincides with the EVERYTHING representation.
+ */
+ private static KeyRange intersectRange(KeyRange a, KeyRange b) {
+ if (a == KeyRange.EVERYTHING_RANGE) {
+ return b;
+ }
+ if (b == KeyRange.EVERYTHING_RANGE) {
+ return a;
+ }
+ return a.intersect(b);
+ }
+
+ /**
+ * Returns the union of {@code this} and {@code other} as a single {@code KeySpace} when
+ * one of the two merge rules applies; otherwise {@link Optional#empty()}.
+ * <ul>
+ * <li>Rule 1 (containment): one space is fully contained in the other; return the larger.</li>
+ * <li>Rule 2 (adjacent boxes): agreeing on all-but-one dim and the remaining dim's ranges
+ * overlap or are adjacent; return the space with the merged dim's range.</li>
+ * </ul>
+ */
+ public Optional<KeySpace> unionIfMergeable(KeySpace other) {
+ requireSameArity(other);
+ if (this.empty) {
+ return Optional.of(other);
+ }
+ if (other.empty) {
+ return Optional.of(this);
+ }
+ if (this.equals(other)) {
+ return Optional.of(this);
+ }
+ if (contains(other)) {
+ return Optional.of(this);
+ }
+ if (other.contains(this)) {
+ return Optional.of(other);
+ }
+ int diffDim = -1;
+ for (int i = 0; i < dims.length; i++) {
+ if (!this.dims[i].equals(other.dims[i])) {
+ if (diffDim != -1) {
+ return Optional.empty();
+ }
+ diffDim = i;
+ }
+ }
+ if (diffDim == -1) {
+ return Optional.of(this);
+ }
+ KeyRange a = this.dims[diffDim];
+ KeyRange b = other.dims[diffDim];
+ // Two distinct single-key points are disjoint and NOT adjacent (they're different
+ // values). KeyRange.intersect has a bug for inverted (DESC) singleton pairs where
+ // the intersection is computed as a non-empty backward range rather than
+ // EMPTY_RANGE, so the check below would incorrectly fall through to union. Detect
+ // this shape explicitly.
+ if (a.isSingleKey() && b.isSingleKey()
+ && !java.util.Arrays.equals(a.getLowerRange(), b.getLowerRange())) {
+ return Optional.empty();
+ }
+ if (a.intersect(b) == KeyRange.EMPTY_RANGE && !isAdjacent(a, b)) {
+ return Optional.empty();
+ }
+ KeyRange[] newDims = dims.clone();
+ newDims[diffDim] = a.union(b);
+ return Optional.of(new KeySpace(newDims, false));
+ }
+
+ /**
+ * Two 1-D ranges are adjacent when the upper bound of one equals the lower bound of the
+ * other and exactly one side is inclusive (so together they cover the shared endpoint
+ * exactly once).
+ */
+ private static boolean isAdjacent(KeyRange a, KeyRange b) {
+ return adjacentOneWay(a, b) || adjacentOneWay(b, a);
+ }
+
+ private static boolean adjacentOneWay(KeyRange a, KeyRange b) {
+ if (a.upperUnbound() || b.lowerUnbound()) {
+ return false;
+ }
+ if (!java.util.Arrays.equals(a.getUpperRange(), b.getLowerRange())) {
+ return false;
+ }
+ return a.isUpperInclusive() != b.isLowerInclusive();
+ }
+
+ /**
+ * {@code this} contains {@code other} iff every dim of {@code this} contains the dim of
+ * {@code other}.
+ */
+ public boolean contains(KeySpace other) {
+ requireSameArity(other);
+ if (other.empty) {
+ return true;
+ }
+ if (this.empty) {
+ return false;
+ }
+ for (int i = 0; i < dims.length; i++) {
+ KeyRange inter = this.dims[i].intersect(other.dims[i]);
+ if (!inter.equals(other.dims[i])) {
+ return false;
+ }
+ }
Review Comment:
Confirmed and fixed. KeyRange.intersect() special-cases IS_NULL to return EMPTY (IS_NULL's empty-byte sentinel clashes with EVERYTHING's UNBOUND), and and() already routed through a private intersectRange() helper to handle that — but contains() was using the raw intersect and would report false for EVERYTHING.contains(IS_NULL). Not a correctness regression (the scan region covers both spaces since no merge happens), but the containment-based merge rule in unionIfMergeable misses the case, leaving the list larger than necessary. Switched contains() to the same helper; added a regression test (containsHandlesEverythingVersusIsNull) that asserts the containment and the subsequent merge.
Note: IS_NOT_NULL doesn't have the same symptom — its lower bound is nextKey(\x00) (not empty), so raw intersect produces a content-equal range and the .equals() check in contains passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192357086
##########
phoenix-core/src/test/java/org/apache/phoenix/compile/WhereCompilerTest.java:
##########
@@ -427,16 +444,13 @@ public void testRowKeyFilter() throws SQLException {
Scan scan = plan.getContext().getScan();
Filter filter = scan.getFilter();
- assertEquals(
- new RowKeyComparisonFilter(
- constantComparison(CompareOperator.EQUAL,
- new SubstrFunction(Arrays.<Expression> asList(
- new RowKeyColumnExpression(ENTITY_ID,
- new RowKeyValueAccessor(ATABLE.getPKColumns(), 1)),
- LiteralExpression.newConstant(1), LiteralExpression.newConstant(3))),
- keyPrefix),
- QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES),
- filter);
Review Comment:
Yes, in test coverage — no, in behavior. I fixed the test in [66a7c45](https://github.com/apache/phoenix/pull/2428/commits/66a7c45d24dad4880064300a207b03e4d7a78811)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-4391459212
> Switching to V2 mode is significant design change. Making that switch by default on 5.4.0 release is something we will have to take it to `dev@phoenix` for consensus. We can only switch to V2 mode smoothly if we know for certain that the V2 mode is 100% compatible with V1. For instance, any phoenix downstream service that relies on explain plan outputs from phoenix cannot expect breaking changes, unless the breaking changes are fixing data integrity issues.
>
> WhereOptimizer test changes seem quite complex to understand. How can we make all tests pass for both V1 and V2 where optimizer if we need lots of assert changes?
>
> Also, I see some tests ignored and some comments like the one with this review that states V2 limitations with some explain plan output. Can we list down all breaking and incompatible changes? We anyways have limited test coverage in where optimizer.
I addressed all your concerns in this commit [d903f1e](https://github.com/apache/phoenix/pull/2428/commits/d903f1e87bfdb8e92d6820837c2ce333168b464a). Please note that the tests are updated only when V2's results is different but equivalent or better. In that case, these tests have different expected results for V1 and V2. The full test suite passes for both V1 and V2. Please note V1 code path will be depreciated and removed in future. That is why V2 is the new default. V1 is preserved to allow rollback when needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5688268524
> @kadirozde Would it be possible for you to go through the Copilot findings and click 'Resolve Conversation' button for everything you think does not apply?
@apurtell Thanks — I’ve gone through your conditional-approval items and the Copilot findings.
Conditional approval items
V2 default off for first ship — done. DEFAULT_WHERE_OPTIMIZER_V2_ENABLED is now false (opt-in via phoenix.where.optimizer.v2.enabled=true). Docs updated accordingly. I’ll also update the PR description, which still incorrectly says “default on.”
collapseToSingleBoundingRange — not removed. Copilot correctly flagged that compounds.subList(0, bound) narrows the union and can drop matching OR branches (false negatives a residual cannot recover). That helper is now used as the over-budget path: collapse to a covering envelope and keep predicates in the residual via the new approximated plumbing. So it is no longer dead code.
DESC separator in buildRvcEqualitySpace — done. Variable-width packed RVC-IN literals now stop on both SEPARATOR_BYTE and DESC_SEPARATOR_BYTE.
KeyRange.hashCode memoization — done. Comment corrected (fields are not final; readFields mutates them), and init() clears cachedHashCode.
Copilot correctness findings
Cartesian bound (allowed = d + 1) — fixed to stop before the overflowing slot (allowed = d), with approximated=true so dropped-slot predicates stay in the residual.
Residual after algebra/extractor approximation — KeySpaceList.isApproximated() / KeyRangeExtractor.Result.approximated / V2ScanBuilder.Result.approximated plumbed through; WhereOptimizerV2 does not strip visitor-consumed nodes when approximated.
Point-lookup classifier accepting leading EVERYTHING — fixed; any unconstrained user dimension rejects the native point-lookup path.
Docs claiming wrong default — fixed with the default-off change.
I’ll click Resolve Conversation on the Copilot threads that are addressed by the above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5703759254
Thanks @kadirozde . All of the Copilot inline threads from the latest review round are resolved.
I checked out the PR head commit and confirmed default is opt-in. Verified each Copilot finding against the patch. Executed a sample of unit tests, all green: WhereOptimizerTest, WhereCompilerTest, QueryOptimizerTest, and the full compile.keyspace.** package.
I do see WhereOptimizerTest.testWithLargeORs fails. It looks like the test wasn't updated after the change pushed up to the PR on Sep 2. I will address this before merging if there are not more issues.
/cc @virajjasani
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3196932830
##########
phoenix-core/src/test/java/org/apache/phoenix/query/QueryPlanTest.java:
##########
@@ -213,6 +221,10 @@ public void testTenantSpecificConnWithLimit() throws Exception {
QueryUtil.getExplainPlan(rs));
}
+ // V2 limitation: DESC timestamp compound byte layout isn't yet decodable by the
+ // ExplainTable splitter (the schema iterator reports mismatched widths for
+ // DESC-inverted timestamp columns).
+ @org.junit.Ignore
Review Comment:
All fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192154208
##########
phoenix-core-client/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java:
##########
@@ -214,6 +216,9 @@ public class QueryServicesOptions {
public static final boolean DEFAULT_IS_NAMESPACE_MAPPING_ENABLED = false;
public static final boolean DEFAULT_IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE = true;
public static final int DEFAULT_MAX_IN_LIST_SKIP_SCAN_SIZE = 50000;
+ public static final boolean DEFAULT_WHERE_OPTIMIZER_V2_ENABLED = true;
Review Comment:
The PR updates a large number of existing test assertions to match V2's behavior rather than keeping the V1 expected output — that's fine when the V2 output is objectively better (tighter scans, correct slotSpan). These do not introduce data integrity issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5823775439
Once this PR review is done, before merging this PR, please do consider turning ON the config to get full test results, and then turn it OFF before PR merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5823654435
I need to rebase this after PHOENIX-7876. Working on that now. /cc @kadirozde
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5841549524
Good merge!
### Test Execution Summary
| Suite / Test File | Type | Mode | Tests | Result |
|---|---|---|---|---|
| [V2ExplainFormatterTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/scan/V2ExplainFormatterTest.java) | UT | V2 | 1 | **PASSED** |
| [KeySpaceTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/KeySpaceTest.java) | UT | V2 | 14 | **PASSED** |
| [KeySpaceListTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/KeySpaceListTest.java) | UT | V2 | 12 | **PASSED** |
| [KeySpaceExpressionVisitorTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/KeySpaceExpressionVisitorTest.java) | UT | V2 | 12 | **PASSED** |
| [KeyRangeExtractorTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/KeyRangeExtractorTest.java) | UT | V2 | 16 | **PASSED** |
| [ExpressionNormalizerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/ExpressionNormalizerTest.java) | UT | V2 | 7 | **PASSED** |
| [DifferentialHarnessTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/algebra/DifferentialHarnessTest.java) | UT | V2 | 5 (5 skp) | **PASSED** |
| [HarnessCorpusTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/algebra/HarnessCorpusTest.java) | UT | V2 | 1 (1 skp) | **PASSED** |
| [ReferenceTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/algebra/ReferenceTest.java) | UT | V2 | 19 | **PASSED** |
| [CompoundByteEncoderTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/scan/CompoundByteEncoderTest.java) | UT | V2 | 5 | **PASSED** |
| [CompoundByteEncoderDifferentialTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/scan/CompoundByteEncoderDifferentialTest.java) | UT | V2 | 9 | **PASSED** |
| [CompoundByteEncoderDescDifferentialTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/scan/CompoundByteEncoderDescDifferentialTest.java) | UT | V2 | 5 | **PASSED** |
| [CompoundByteEncoderListDifferentialTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/scan/CompoundByteEncoderListDifferentialTest.java) | UT | V2 | 6 | **PASSED** |
| [RVCScanBoundariesEncoderTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/keyspace/scan/RVCScanBoundariesEncoderTest.java) | UT | V2 | 1 | **PASSED** |
| [ExplainPlanTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/query/explain/ExplainPlanTest.java) | UT | Default | 111 | **PASSED** |
| [ExplainJsonOutputTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/query/explain/ExplainJsonOutputTest.java) | UT | Default | 2 | **PASSED** |
| [ExplainOptionsParserTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/parse/ExplainOptionsParserTest.java) | UT | Default | 16 | **PASSED** |
| [QueryPlanTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/query/QueryPlanTest.java) | UT | Default | 4 | **PASSED** |
| [QueryCompilerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/QueryCompilerTest.java) | UT | Default | 215 (2 skp) | **PASSED** |
| [QueryOptimizerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/QueryOptimizerTest.java) | UT | Default | 53 (1 skp) | **PASSED** |
| [StatementHintsCompilationTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/StatementHintsCompilationTest.java) | UT | Default | 4 | **PASSED** |
| [TenantSpecificViewIndexCompileTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/TenantSpecificViewIndexCompileTest.java) | UT | Default | 6 | **PASSED** |
| [JoinQueryCompilerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/JoinQueryCompilerTest.java) | UT | Default | 2 | **PASSED** |
| [WhereCompilerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/WhereCompilerTest.java) | UT | Default / V2 | 46 (1 skp) | **PASSED** |
| [WhereOptimizerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/WhereOptimizerTest.java) | UT | V2 (`-D...v2.enabled=true`) | 142 | **PASSED** |
| [WhereOptimizerTest](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/test/java/org/apache/phoenix/compile/WhereOptimizerTest.java) | UT | Baseline (V1) | 142 | **PASSED** |
| [ExplainPlanWithStatsDisabledIT](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/it/java/org/apache/phoenix/end2end/ExplainPlanWithStatsDisabledIT.java) | IT | Cluster | 17 | **PASSED** |
| [ExplainPlanWithStatsEnabledIT](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/it/java/org/apache/phoenix/end2end/ExplainPlanWithStatsEnabledIT.java) | IT | Cluster | 30 | **PASSED** |
| [WhereOptimizerForArrayAnyIT](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/it/java/org/apache/phoenix/end2end/WhereOptimizerForArrayAnyIT.java) | IT | Cluster | 20 | **PASSED** |
| [WhereOptimizerForArrayAnyNullablePKIT](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/it/java/org/apache/phoenix/end2end/WhereOptimizerForArrayAnyNullablePKIT.java) | IT | Cluster | 96 (16 skp) | **PASSED** |
| [WhereOptimizerForJoinFiltersIT](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/it/java/org/apache/phoenix/end2end/join/WhereOptimizerForJoinFiltersIT.java) | IT | Cluster | 1 | **PASSED** |
| [CostBasedDecisionIT](file:///Users/andrewpurtell/src/phoenix/phoenix-core/src/it/java/org/apache/phoenix/end2end/CostBasedDecisionIT.java) | IT | Cluster | 20 | **PASSED** |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192229704
##########
phoenix-core-client/src/main/java/org/apache/phoenix/optimize/QueryOptimizer.java:
##########
@@ -792,6 +792,76 @@ public int compare(QueryPlan plan1, QueryPlan plan2) {
return stopAtBestPlan ? bestCandidates.subList(0, 1) : bestCandidates;
}
+ /**
+ * Variant of {@link org.apache.phoenix.compile.ScanRanges#getBoundPkColumnCount} for the cost
+ * comparator: only honors the extra span reported by {@code slotSpan[i]} when the slot's ranges
+ * genuinely span multiple PK columns via byte-level encoding. A slot whose only ranges are
+ * non-point finite-bounded scalars (e.g. {@code BETWEEN a AND b} on a single PK col) under V2
+ * can carry an artificial {@code slotSpan[i] > 0} set by
+ * {@link org.apache.phoenix.compile.keyspace.KeyRangeExtractor#emitV1Projection} as a
+ * V1-compat marker for {@link org.apache.phoenix.filter.SkipScanFilter}'s per-region cursor
+ * stepping. That marker is needed for SkipScanFilter correctness on CDC-style scans but
+ * would inflate the PK-col count the cost comparator ranks plans by, letting a plan with a
+ * narrow bounded range on one PK col tie with a plan whose compound range genuinely covers
+ * multiple PK cols. Tie on primary key falls through to weaker tiebreakers that flip the
+ * plan choice (see CostBasedDecisionIT.testCostOverridesStaticPlanOrdering3 and its
+ * Upsert/Delete variants).
+ * <p>
+ * Rule: {@code slotSpan[i]} is honored unless every range in the slot is a non-point
+ * finite-bounded scalar. Point keys (single-key compound bytes encoding multiple PK cols,
+ * e.g. from RVC-IN or tuple equality) and ranges with an UNBOUND side (where trailing cols
+ * inherit the unbounded side's min/max at the byte level) both represent genuine multi-col
+ * narrowing and are counted as {@code slotSpan[i]+1}.
+ */
+ private static int effectiveBoundPkColumnCount(
+ org.apache.phoenix.compile.ScanRanges scanRanges) {
+ java.util.List<java.util.List<org.apache.phoenix.query.KeyRange>> ranges =
+ scanRanges.getRanges();
+ int[] slotSpan = scanRanges.getSlotSpans();
+ int count = 0;
+ boolean hasUnbound = false;
+ int nRanges = ranges.size();
+ for (int i = 0; i < nRanges && !hasUnbound; i++) {
+ java.util.List<org.apache.phoenix.query.KeyRange> orRanges = ranges.get(i);
+ boolean slotHasUnbound = false;
+ boolean slotAllBoundedNonPoint = true;
+ for (org.apache.phoenix.query.KeyRange range : orRanges) {
+ if (range == org.apache.phoenix.query.KeyRange.EVERYTHING_RANGE) {
+ return count;
+ }
+ if (range.isUnbound()) {
+ slotHasUnbound = true;
+ hasUnbound = true;
+ }
+ if (range.isSingleKey() || range.isUnbound()) {
+ slotAllBoundedNonPoint = false;
+ }
+ }
+ int span = (slotSpan != null && i < slotSpan.length) ? slotSpan[i] : 0;
+ // V2-only artificial extension: {@code KeyRangeExtractor.emitV1Projection} bumps
+ // {@code slotSpan[last]} to span trailing unconstrained PK cols for
+ // {@link SkipScanFilter} cursor stepping, even when the slot holds a single-column
+ // finite-bounded scalar range whose bytes don't actually encode trailing cols.
+ // For cost ranking we want to count only columns genuinely narrowed by the scan
+ // bounds, so discount that shape to 1.
+ //
+ // Discriminator: the extension path ALWAYS has >=2 slots (at least one leading
+ // concretely-narrowed slot plus the last bumped slot). A single-slot shape with
+ // {@code slotSpan > 0} is the compound-emission path — a genuine multi-col range
+ // whose bytes concatenate {@code span+1} columns — and keeps {@code span + 1}.
+ // See CostBasedDecisionIT.testCostOverridesStaticPlanOrdering3 (multi-slot shape
+ // must discount) vs. RowTimestampIT.testAutomaticallySettingRowTimestampWith*
+ // (single-slot compound must not discount).
+ boolean isArtificialExtension = nRanges > 1 && i == nRanges - 1;
Review Comment:
Good catch on the single-slot extension case — the nRanges > 1 guard was too coarse and would have over-counted it. But i == nRanges - 1 alone would re-introduce a different regression: single-slot compound ranges (RowTimestampIT.testAutomaticallySettingRowTimestampWith* — a PK1=v AND PK2 BETWEEN a AND b data-table plan emits one slot with slotSpan=[1] and a compound range concatenating PK1's bytes + separator + PK2's bytes) are genuine 2-col narrowing that must count as span+1, not 1.
The distinguishing signal is byte-vs-schema: a compound range's bytes decode into span+1 PK fields; a V1-compat extension's bytes cover one field. I switched the discriminator to a byte-length decode against the schema so both cases resolve correctly:
Multi-slot extension (CostBasedDecisionIT.testCostOverridesStaticPlanOrdering3): bytes cover 1 field → discount to 1.
Single-slot extension (your new case): bytes cover 1 field → discount to 1.
Single-slot compound (RowTimestampIT): bytes cover span+1 fields → keep span+1.
Point-key compound (testQueryOptimizerShouldSelectThePlanWithMoreNumberOfPKColumns): gated out by slotAllBoundedNonPoint=false, falls through to span+1.
Verified: RowTimestampIT 24/24, CostBasedDecisionIT 20/20, WhereOptimizerTest 142/142, QueryOptimizerTest 48/48.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3190918532
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/oracle/Oracle.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.oracle;
+
+/**
+ * Reference implementation (oracle) of the key-space model's key-range extraction
+ * algorithm. Given an {@link AbstractExpression} tree over a schema with {@code nPk}
+ * primary-key dimensions, produces the {@link AbstractKeySpaceList} the algorithm
+ * should emit.
+ * <p>
+ * The purpose is differential testing: we compare the oracle's output against the
+ * production {@code WhereOptimizerV2} implementation's {@code KeySpaceList} to detect
+ * divergences. Any difference is either a production bug or an oracle bug — the oracle
+ * being shorter and directly derived from the design, the default suspect is production.
+ * <p>
+ * This oracle does not handle:
+ * <ul>
+ * <li>Normalization of RVC inequalities (feed in the lex-expanded form).</li>
+ * <li>Byte encoding, DESC inversion, separator bytes, salt/tenant prefixes.</li>
+ * <li>Null handling (IS NULL / IS NOT NULL).</li>
+ * <li>Scalar function wrappers or coercions.</li>
+ * </ul>
+ * All of those are production concerns that live above the algebra the model describes.
+ * <p>
+ * <b>Correctness property.</b> For every row {@code r} where
+ * {@code expr.evaluate(r) == true}, the emitted {@link AbstractKeySpaceList} must match
+ * {@code r} (soundness: no false negatives). False positives — rows in the list but not
+ * satisfying the expression — are permitted because the production residual filter
+ * re-evaluates the original predicate at scan time.
+ */
+public final class Oracle {
+
+ private Oracle() {}
Review Comment:
I don't think this and relevant classes should be part of our source code. Any comparison with Oracle or Postgresql should be only part of the test code. I also think Postgres would be better choice as it is open source, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3190968323
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/oracle/Oracle.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.oracle;
+
+/**
+ * Reference implementation (oracle) of the key-space model's key-range extraction
+ * algorithm. Given an {@link AbstractExpression} tree over a schema with {@code nPk}
+ * primary-key dimensions, produces the {@link AbstractKeySpaceList} the algorithm
+ * should emit.
+ * <p>
+ * The purpose is differential testing: we compare the oracle's output against the
+ * production {@code WhereOptimizerV2} implementation's {@code KeySpaceList} to detect
+ * divergences. Any difference is either a production bug or an oracle bug — the oracle
+ * being shorter and directly derived from the design, the default suspect is production.
+ * <p>
+ * This oracle does not handle:
+ * <ul>
+ * <li>Normalization of RVC inequalities (feed in the lex-expanded form).</li>
+ * <li>Byte encoding, DESC inversion, separator bytes, salt/tenant prefixes.</li>
+ * <li>Null handling (IS NULL / IS NOT NULL).</li>
+ * <li>Scalar function wrappers or coercions.</li>
+ * </ul>
+ * All of those are production concerns that live above the algebra the model describes.
+ * <p>
+ * <b>Correctness property.</b> For every row {@code r} where
+ * {@code expr.evaluate(r) == true}, the emitted {@link AbstractKeySpaceList} must match
+ * {@code r} (soundness: no false negatives). False positives — rows in the list but not
+ * satisfying the expression — are permitted because the production residual filter
+ * re-evaluates the original predicate at scan time.
+ */
+public final class Oracle {
+
+ private Oracle() {}
Review Comment:
Here Oracle means the reference implementation for the key space algebra, not the Oracle database. I can rename it and also move it to the test package as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192389897
##########
phoenix-core-client/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java:
##########
@@ -214,6 +216,9 @@ public class QueryServicesOptions {
public static final boolean DEFAULT_IS_NAMESPACE_MAPPING_ENABLED = false;
public static final boolean DEFAULT_IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE = true;
public static final int DEFAULT_MAX_IN_LIST_SKIP_SCAN_SIZE = 50000;
+ public static final boolean DEFAULT_WHERE_OPTIMIZER_V2_ENABLED = true;
Review Comment:
I just noticed that there were 18 ignored tests for V2. I am working on them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5828177727
@kadirozde I have merged this branch with the latest HEAD:
https://github.com/apurtell/phoenix/tree/PHOENIX-6791
--> https://github.com/apurtell/phoenix/commits/PHOENIX-6791
--> --> https://github.com/apurtell/phoenix/commit/bde2056c454cdabd40c94842b22567022d4a6fa8
If you can give me collaborator access to https://github.com/kadirozde/phoenix temporarily, I may be able to push the changes directly to this PR. Otherwise I can open a new PR with that for merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192177935
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/StatementContext.java:
##########
@@ -309,6 +310,20 @@ public void setScanRanges(ScanRanges scanRanges) {
scanRanges.initializeScan(scan);
}
+ /**
+ * V2-owned metadata attached by the V2 scan-construction path; null under the V1 path
+ * ({@code WHERE_OPTIMIZER_V2_ENABLED=false}). Consumers (currently the explain-plan
+ * formatter) prefer this when present; others are unaffected.
+ */
+ public org.apache.phoenix.compile.keyspace.scan.V2ScanArtifact getV2ScanArtifact() {
+ return this.v2ScanArtifact;
+ }
+
+ public void setV2ScanArtifact(
+ org.apache.phoenix.compile.keyspace.scan.V2ScanArtifact artifact) {
+ this.v2ScanArtifact = artifact;
Review Comment:
Fixed it with commit 9769d0e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3190887178
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/StatementContext.java:
##########
@@ -309,6 +310,20 @@ public void setScanRanges(ScanRanges scanRanges) {
scanRanges.initializeScan(scan);
}
+ /**
+ * V2-owned metadata attached by the V2 scan-construction path; null under the V1 path
+ * ({@code WHERE_OPTIMIZER_V2_ENABLED=false}). Consumers (currently the explain-plan
+ * formatter) prefer this when present; others are unaffected.
+ */
+ public org.apache.phoenix.compile.keyspace.scan.V2ScanArtifact getV2ScanArtifact() {
+ return this.v2ScanArtifact;
+ }
+
+ public void setV2ScanArtifact(
+ org.apache.phoenix.compile.keyspace.scan.V2ScanArtifact artifact) {
+ this.v2ScanArtifact = artifact;
Review Comment:
scanRanges and v2ScanArtifact can now diverge because setScanRanges() never clears or updates the artifact. Any later setScanRanges(...) on the same StatementContext—such as switching to EVERYTHING/NOTHING or replacing the scan shape during planning—will leave ExplainTable formatting key ranges from stale V2 metadata.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/optimize/QueryOptimizer.java:
##########
@@ -792,6 +792,76 @@ public int compare(QueryPlan plan1, QueryPlan plan2) {
return stopAtBestPlan ? bestCandidates.subList(0, 1) : bestCandidates;
}
+ /**
+ * Variant of {@link org.apache.phoenix.compile.ScanRanges#getBoundPkColumnCount} for the cost
+ * comparator: only honors the extra span reported by {@code slotSpan[i]} when the slot's ranges
+ * genuinely span multiple PK columns via byte-level encoding. A slot whose only ranges are
+ * non-point finite-bounded scalars (e.g. {@code BETWEEN a AND b} on a single PK col) under V2
+ * can carry an artificial {@code slotSpan[i] > 0} set by
+ * {@link org.apache.phoenix.compile.keyspace.KeyRangeExtractor#emitV1Projection} as a
+ * V1-compat marker for {@link org.apache.phoenix.filter.SkipScanFilter}'s per-region cursor
+ * stepping. That marker is needed for SkipScanFilter correctness on CDC-style scans but
+ * would inflate the PK-col count the cost comparator ranks plans by, letting a plan with a
+ * narrow bounded range on one PK col tie with a plan whose compound range genuinely covers
+ * multiple PK cols. Tie on primary key falls through to weaker tiebreakers that flip the
+ * plan choice (see CostBasedDecisionIT.testCostOverridesStaticPlanOrdering3 and its
+ * Upsert/Delete variants).
+ * <p>
+ * Rule: {@code slotSpan[i]} is honored unless every range in the slot is a non-point
+ * finite-bounded scalar. Point keys (single-key compound bytes encoding multiple PK cols,
+ * e.g. from RVC-IN or tuple equality) and ranges with an UNBOUND side (where trailing cols
+ * inherit the unbounded side's min/max at the byte level) both represent genuine multi-col
+ * narrowing and are counted as {@code slotSpan[i]+1}.
+ */
+ private static int effectiveBoundPkColumnCount(
+ org.apache.phoenix.compile.ScanRanges scanRanges) {
+ java.util.List<java.util.List<org.apache.phoenix.query.KeyRange>> ranges =
+ scanRanges.getRanges();
+ int[] slotSpan = scanRanges.getSlotSpans();
+ int count = 0;
+ boolean hasUnbound = false;
+ int nRanges = ranges.size();
+ for (int i = 0; i < nRanges && !hasUnbound; i++) {
+ java.util.List<org.apache.phoenix.query.KeyRange> orRanges = ranges.get(i);
+ boolean slotHasUnbound = false;
+ boolean slotAllBoundedNonPoint = true;
+ for (org.apache.phoenix.query.KeyRange range : orRanges) {
+ if (range == org.apache.phoenix.query.KeyRange.EVERYTHING_RANGE) {
+ return count;
+ }
+ if (range.isUnbound()) {
+ slotHasUnbound = true;
+ hasUnbound = true;
+ }
+ if (range.isSingleKey() || range.isUnbound()) {
+ slotAllBoundedNonPoint = false;
+ }
+ }
+ int span = (slotSpan != null && i < slotSpan.length) ? slotSpan[i] : 0;
+ // V2-only artificial extension: {@code KeyRangeExtractor.emitV1Projection} bumps
+ // {@code slotSpan[last]} to span trailing unconstrained PK cols for
+ // {@link SkipScanFilter} cursor stepping, even when the slot holds a single-column
+ // finite-bounded scalar range whose bytes don't actually encode trailing cols.
+ // For cost ranking we want to count only columns genuinely narrowed by the scan
+ // bounds, so discount that shape to 1.
+ //
+ // Discriminator: the extension path ALWAYS has >=2 slots (at least one leading
+ // concretely-narrowed slot plus the last bumped slot). A single-slot shape with
+ // {@code slotSpan > 0} is the compound-emission path — a genuine multi-col range
+ // whose bytes concatenate {@code span+1} columns — and keeps {@code span + 1}.
+ // See CostBasedDecisionIT.testCostOverridesStaticPlanOrdering3 (multi-slot shape
+ // must discount) vs. RowTimestampIT.testAutomaticallySettingRowTimestampWith*
+ // (single-slot compound must not discount).
+ boolean isArtificialExtension = nRanges > 1 && i == nRanges - 1;
Review Comment:
The 'ALWAYS has >=2 slots' assumption is too strong. The V1-compat slotSpan extension can also appear when there is only one emitted range slot followed by unconstrained trailing PK columns, so single-slot plans can still be artificial here. Those scans will be over-counted as genuinely multi-column and can still skew plan ranking.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/oracle/Oracle.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.oracle;
+
+/**
+ * Reference implementation (oracle) of the key-space model's key-range extraction
+ * algorithm. Given an {@link AbstractExpression} tree over a schema with {@code nPk}
+ * primary-key dimensions, produces the {@link AbstractKeySpaceList} the algorithm
+ * should emit.
+ * <p>
+ * The purpose is differential testing: we compare the oracle's output against the
+ * production {@code WhereOptimizerV2} implementation's {@code KeySpaceList} to detect
+ * divergences. Any difference is either a production bug or an oracle bug — the oracle
+ * being shorter and directly derived from the design, the default suspect is production.
Review Comment:
This package is explicitly described as differential-test scaffolding, but it is being added under src/main/java, which ships it in the client artifact and turns the reference model into production package surface. These classes should live under test sources or a dedicated test jar instead of the runtime module.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/KeySpace.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace;
+
+import java.util.Arrays;
+import java.util.Optional;
+
+import org.apache.phoenix.query.KeyRange;
+
+/**
+ * An N-dimensional key space over a table's primary key columns. Each dimension is a
+ * {@link KeyRange} over the encoded byte representation of a single PK column. An
+ * expression node is modeled as a list of {@code KeySpace} instances; see
+ * {@link KeySpaceList} for the list-level algebra.
+ * <p>
+ * Instances are immutable. {@link #and(KeySpace)} is the per-dimension intersection;
+ * {@link #unionIfMergeable(KeySpace)} returns the union when either (a) one space contains
+ * the other or (b) the two spaces agree on all but one dimension and the differing dim's
+ * ranges are non-disjoint.
+ * <p>
+ * A single-value predicate like {@code PK2 >= 3} on a 3-PK table is represented as
+ * {@code [(*,*), [3,*), (*,*)]} — a singleton {@link KeySpaceList} containing a single
+ * {@code KeySpace} where every dim not mentioned in the predicate holds
+ * {@link KeyRange#EVERYTHING_RANGE}. RVC inequalities are pre-normalized by
+ * {@link ExpressionNormalizer} into lexicographic AND/OR of scalar comparisons so that
+ * per-dim intersection composes correctly with every other predicate; this class therefore
+ * never needs to model compound-byte concatenation directly.
+ */
+public final class KeySpace {
+
+ private final KeyRange[] dims;
+ private final boolean empty;
+
+ private KeySpace(KeyRange[] dims, boolean empty) {
+ this.dims = dims;
+ this.empty = empty;
+ }
+
+ public static KeySpace everything(int n) {
+ KeyRange[] dims = new KeyRange[n];
+ Arrays.fill(dims, KeyRange.EVERYTHING_RANGE);
+ return new KeySpace(dims, false);
+ }
+
+ public static KeySpace empty(int n) {
+ KeyRange[] dims = new KeyRange[n];
+ Arrays.fill(dims, KeyRange.EMPTY_RANGE);
+ return new KeySpace(dims, true);
+ }
+
+ public static KeySpace single(int dim, KeyRange r, int n) {
+ if (r == KeyRange.EMPTY_RANGE) {
+ return empty(n);
+ }
+ KeyRange[] dims = new KeyRange[n];
+ Arrays.fill(dims, KeyRange.EVERYTHING_RANGE);
+ dims[dim] = r;
+ return new KeySpace(dims, false);
+ }
+
+ public static KeySpace of(KeyRange[] dims) {
+ boolean empty = false;
+ for (KeyRange r : dims) {
+ if (r == KeyRange.EMPTY_RANGE) {
+ empty = true;
+ break;
+ }
+ }
+ return new KeySpace(dims.clone(), empty);
+ }
+
+ public int nDims() {
+ return dims.length;
+ }
+
+ public KeyRange get(int dim) {
+ return dims[dim];
+ }
+
+ /**
+ * Returns a new {@link KeySpace} identical to this one except with dim {@code dim}
+ * replaced by {@code r}. Used by {@link KeySpaceList}'s widening path to drop a
+ * trailing dim (by replacing it with {@link KeyRange#EVERYTHING_RANGE}). Allocates a
+ * fresh dims array; original is unchanged.
+ */
+ public KeySpace withDimReplaced(int dim, KeyRange r) {
+ if (dims[dim].equals(r)) {
+ return this;
+ }
+ KeyRange[] newDims = dims.clone();
+ newDims[dim] = r;
+ if (r == KeyRange.EMPTY_RANGE) {
+ return empty(dims.length);
+ }
+ return new KeySpace(newDims, false);
+ }
+
+ public boolean isEmpty() {
+ return empty;
+ }
+
+ public boolean isEverything() {
+ if (empty) {
+ return false;
+ }
+ for (KeyRange r : dims) {
+ if (r != KeyRange.EVERYTHING_RANGE) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Per-dimension intersection. If any dim collapses to {@link KeyRange#EMPTY_RANGE}, the
+ * result is {@link #empty(int)}.
+ */
+ public KeySpace and(KeySpace other) {
+ requireSameArity(other);
+ if (this.empty || other.empty) {
+ return empty(dims.length);
+ }
+ KeyRange[] newDims = new KeyRange[dims.length];
+ for (int i = 0; i < dims.length; i++) {
+ KeyRange a = this.dims[i];
+ KeyRange b = other.dims[i];
+ KeyRange inter = intersectRange(a, b);
+ if (inter == KeyRange.EMPTY_RANGE) {
+ return empty(dims.length);
+ }
+ newDims[i] = inter;
+ }
+ return new KeySpace(newDims, false);
+ }
+
+ /**
+ * Per-dim intersection that special-cases {@link KeyRange#EVERYTHING_RANGE} against
+ * {@link KeyRange#IS_NULL_RANGE} / {@link KeyRange#IS_NOT_NULL_RANGE}. Plain
+ * {@link KeyRange#intersect} treats EVERYTHING ∩ IS_NULL as EMPTY because IS_NULL uses
+ * an empty-byte-array sentinel that coincides with the EVERYTHING representation.
+ */
+ private static KeyRange intersectRange(KeyRange a, KeyRange b) {
+ if (a == KeyRange.EVERYTHING_RANGE) {
+ return b;
+ }
+ if (b == KeyRange.EVERYTHING_RANGE) {
+ return a;
+ }
+ return a.intersect(b);
+ }
+
+ /**
+ * Returns the union of {@code this} and {@code other} as a single {@code KeySpace} when
+ * one of the two merge rules applies; otherwise {@link Optional#empty()}.
+ * <ul>
+ * <li>Rule 1 (containment): one space is fully contained in the other; return the larger.</li>
+ * <li>Rule 2 (adjacent boxes): agreeing on all-but-one dim and the remaining dim's ranges
+ * overlap or are adjacent; return the space with the merged dim's range.</li>
+ * </ul>
+ */
+ public Optional<KeySpace> unionIfMergeable(KeySpace other) {
+ requireSameArity(other);
+ if (this.empty) {
+ return Optional.of(other);
+ }
+ if (other.empty) {
+ return Optional.of(this);
+ }
+ if (this.equals(other)) {
+ return Optional.of(this);
+ }
+ if (contains(other)) {
+ return Optional.of(this);
+ }
+ if (other.contains(this)) {
+ return Optional.of(other);
+ }
+ int diffDim = -1;
+ for (int i = 0; i < dims.length; i++) {
+ if (!this.dims[i].equals(other.dims[i])) {
+ if (diffDim != -1) {
+ return Optional.empty();
+ }
+ diffDim = i;
+ }
+ }
+ if (diffDim == -1) {
+ return Optional.of(this);
+ }
+ KeyRange a = this.dims[diffDim];
+ KeyRange b = other.dims[diffDim];
+ // Two distinct single-key points are disjoint and NOT adjacent (they're different
+ // values). KeyRange.intersect has a bug for inverted (DESC) singleton pairs where
+ // the intersection is computed as a non-empty backward range rather than
+ // EMPTY_RANGE, so the check below would incorrectly fall through to union. Detect
+ // this shape explicitly.
+ if (a.isSingleKey() && b.isSingleKey()
+ && !java.util.Arrays.equals(a.getLowerRange(), b.getLowerRange())) {
+ return Optional.empty();
+ }
+ if (a.intersect(b) == KeyRange.EMPTY_RANGE && !isAdjacent(a, b)) {
+ return Optional.empty();
+ }
+ KeyRange[] newDims = dims.clone();
+ newDims[diffDim] = a.union(b);
+ return Optional.of(new KeySpace(newDims, false));
+ }
+
+ /**
+ * Two 1-D ranges are adjacent when the upper bound of one equals the lower bound of the
+ * other and exactly one side is inclusive (so together they cover the shared endpoint
+ * exactly once).
+ */
+ private static boolean isAdjacent(KeyRange a, KeyRange b) {
+ return adjacentOneWay(a, b) || adjacentOneWay(b, a);
+ }
+
+ private static boolean adjacentOneWay(KeyRange a, KeyRange b) {
+ if (a.upperUnbound() || b.lowerUnbound()) {
+ return false;
+ }
+ if (!java.util.Arrays.equals(a.getUpperRange(), b.getLowerRange())) {
+ return false;
+ }
+ return a.isUpperInclusive() != b.isLowerInclusive();
+ }
+
+ /**
+ * {@code this} contains {@code other} iff every dim of {@code this} contains the dim of
+ * {@code other}.
+ */
+ public boolean contains(KeySpace other) {
+ requireSameArity(other);
+ if (other.empty) {
+ return true;
+ }
+ if (this.empty) {
+ return false;
+ }
+ for (int i = 0; i < dims.length; i++) {
+ KeyRange inter = this.dims[i].intersect(other.dims[i]);
+ if (!inter.equals(other.dims[i])) {
+ return false;
+ }
+ }
Review Comment:
contains() still relies on raw KeyRange.intersect(), even though and() already had to special-case EVERYTHING with IS_NULL/IS_NOT_NULL because intersect() collapses that combination to EMPTY. A KeySpace containing EVERYTHING will therefore incorrectly report that it does not contain null/not-null ranges, which breaks OR-merge containment for those predicates.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/query/QueryServicesOptions.java:
##########
@@ -214,6 +216,9 @@ public class QueryServicesOptions {
public static final boolean DEFAULT_IS_NAMESPACE_MAPPING_ENABLED = false;
public static final boolean DEFAULT_IS_SYSTEM_TABLE_MAPPED_TO_NAMESPACE = true;
public static final int DEFAULT_MAX_IN_LIST_SKIP_SCAN_SIZE = 50000;
+ public static final boolean DEFAULT_WHERE_OPTIMIZER_V2_ENABLED = true;
Review Comment:
This makes the new optimizer opt-out for every deployment, but the same PR is still disabling a large number of existing tests for known V2 gaps (explain parity, ORDER BY/GROUP BY optimization, and index-selection behavior). Enabling it by default will expose those known regressions in production; the default should stay false until those cases are no longer being skipped.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/ExpressionNormalizer.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace;
+
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hbase.CompareOperator;
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.expression.AndExpression;
+import org.apache.phoenix.expression.ComparisonExpression;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.InListExpression;
+import org.apache.phoenix.expression.OrExpression;
+import org.apache.phoenix.expression.RowValueConstructorExpression;
+
+/**
+ * Rewrites a WHERE {@link Expression} into the canonical AND/OR form the v2 key-space model
+ * operates on. Two rewrites are applied bottom-up:
+ * <ul>
+ * <li><b>RVC inequality</b>: {@code (c1,...,cK) OP (v1,...,vK)} for OP in
+ * {@code <, ≤, >, ≥} expands to the lexicographic OR of ANDs. Example:
+ * {@code (c1,c2,c3) > (v1,v2,v3)} becomes
+ * {@code (c1>v1) OR (c1=v1 AND c2>v2) OR (c1=v1 AND c2=v2 AND c3>v3)}. Equality is already
+ * expanded upstream by {@link ComparisonExpression#create}.</li>
+ * <li><b>IN list on a scalar</b>: {@code a IN (v1,...,vK)} expands to
+ * {@code a=v1 OR ... OR a=vK}. IN with an RVC LHS is left intact; the visitor handles
+ * that shape directly by producing one KeySpace per row value.</li>
+ * </ul>
Review Comment:
The class-level contract no longer matches the implementation. Scalar IN lists are now left intact because rewriteScalarInList() is a documented no-op, but the Javadoc still says they are rewritten into OR-of-equalities.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/scan/V2ExplainFormatter.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.scan;
+
+import java.text.Format;
+
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.compile.StatementContext;
+import org.apache.phoenix.compile.keyspace.KeySpace;
+import org.apache.phoenix.compile.keyspace.KeySpaceList;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.SortOrder;
+import org.apache.phoenix.schema.TableRef;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.StringUtil;
+
+/**
+ * V2-owned explain-plan keyRanges formatter.
+ * <p>
+ * Reads from {@link V2ScanArtifact#list()} directly rather than re-decoding the byte-
+ * encoded {@code ScanRanges} that drives actual scan execution. The artifact carries the
+ * pre-encoding, mathematical form of the scan, so inclusive-upper displays as
+ * {@code [*, 1]} (matching V1) instead of {@code [*, 2)} (what V2's compound byte
+ * emission produces after {@code nextKey(1) = 2}).
+ * <p>
+ * Current scope: single-space KeySpaceList with point or range ranges per dim. For
+ * multi-space lists the caller falls back to the legacy byte-decoding path in
+ * {@code ExplainTable.appendKeyRanges}, which produces different but semantically
+ * equivalent output. A future extension will handle the multi-space case by computing
+ * per-dim unions and emitting SkipScanFilter-style displays.
+ */
+public final class V2ExplainFormatter {
+
+ private V2ExplainFormatter() {
+ }
+
+ /**
+ * Build the keyRanges display string for a scan whose V2 artifact is {@code artifact}.
+ * Returns {@code null} if this formatter does not handle the input shape — the caller
+ * should fall back to the legacy formatter.
+ */
+ public static String appendKeyRanges(StatementContext context, TableRef tableRef,
+ V2ScanArtifact artifact) {
+ KeySpaceList list = artifact.list();
+ if (list.isUnsatisfiable()) {
+ return "";
+ }
+ PTable table = tableRef.getTable();
+ int nPk = artifact.nPkColumns();
+ int prefixSlots = artifact.prefixSlots();
+
+ // KeySpaceList.isEverything() means no user-dim constraint, but when prefix slots
+ // (salt / viewIndexId / tenantId) are present the scan is still narrowed to the
+ // tenant partition — render those prefix values. Without this, tenant-only queries
+ // display as empty brackets while V1 shows {@code ['tenantId']}.
+ if (list.isEverything()) {
+ return renderPrefixOnly(context, table, prefixSlots);
+ }
+ if (list.size() != 1) {
+ // Multi-space path not implemented yet; fall back.
+ return null;
+ }
+ KeySpace space = list.spaces().get(0);
+ boolean isLocalIndex = org.apache.phoenix.util.ScanUtil.isLocalIndex(context.getScan());
+
+ // KeySpace indexing is by absolute PK position ({@link KeySpaceExpressionVisitor}
+ // emits ranges via {@code KeySpace.single(pkPos, ...)} with {@code pkPos} being the
+ // column's absolute index in the PK). So {@code space.get(d)} — where d is an
+ // absolute PK index — returns the dim for PK column d. Prefix dims (viewIndexId /
+ // tenantId / salt) are always EVERYTHING inside the KeySpaceList because the visitor
+ // doesn't know about them; prefix values are rendered from scanRanges below.
+ //
+ // Fall back when a dim's raw bytes don't match the PK column's full fixed-width size.
+ // Happens when a scalar function (SUBSTR, FLOOR, ...) creates a KeyRange with truncated
+ // bytes (e.g. 3-byte substr prefix on an 8-byte LONG column). V1's ExplainTable reads
+ // post-processing ScanRanges bytes which have been zero-padded to the field width; V2's
+ // artifact carries the raw (pre-processing) bytes. Rather than duplicating V1's padding
+ // logic, defer to the legacy byte-decoding formatter.
+ for (int d = prefixSlots; d < nPk && d < space.nDims(); d++) {
+ KeyRange kr = space.get(d);
+ if (kr == KeyRange.EVERYTHING_RANGE || kr == KeyRange.IS_NULL_RANGE
+ || kr == KeyRange.IS_NOT_NULL_RANGE) {
+ continue;
+ }
+ PColumn col = table.getPKColumns().get(d);
+ PDataType type = col.getDataType();
+ if (!type.isFixedWidth()) {
+ continue;
+ }
+ Integer maxLen = col.getMaxLength();
+ Integer typeSize = type.getByteSize();
+ if (maxLen == null && typeSize == null) {
+ continue;
+ }
+ int expected = maxLen != null ? maxLen : typeSize;
+ byte[] lb = kr.getRange(KeyRange.Bound.LOWER);
+ byte[] ub = kr.getRange(KeyRange.Bound.UPPER);
+ if ((lb != null && lb.length != 0 && lb.length != expected)
+ || (ub != null && ub.length != 0 && ub.length != expected)) {
+ return null;
+ }
+ }
+
+ // Last-dim-to-display index: highest dim with a non-EVERYTHING range. V1 truncates
+ // at the first EVERYTHING past the prefix so the display doesn't end in `*,*,*`.
+ int lastConstrained = prefixSlots - 1;
+ for (int d = prefixSlots; d < nPk && d < space.nDims(); d++) {
+ KeyRange kr = space.get(d);
+ if (kr != KeyRange.EVERYTHING_RANGE) {
+ lastConstrained = d;
+ }
+ }
+ if (lastConstrained < prefixSlots) {
+ return "";
+ }
+
+ StringBuilder lower = new StringBuilder();
+ StringBuilder upper = new StringBuilder();
+ // Prefix columns (salt byte / viewIndexId / tenantId) aren't in the KeySpaceList —
+ // they're auto-populated slots. Read them from the ScanRanges's per-slot structure,
+ // which the caller already built. This keeps prefix display identical to V1's.
+ int prefixEmitted = 0;
+ if (context.getScanRanges() != null && !context.getScanRanges().getRanges().isEmpty()) {
+ java.util.List<java.util.List<KeyRange>> ranges = context.getScanRanges().getRanges();
+ for (int d = 0; d < prefixSlots && d < ranges.size(); d++) {
+ // Slot 0 on a salted table holds the full [0..nBuckets-1] salt-bucket range
+ // list after {@link ScanRanges}'s constructor populates it via
+ // {@link SaltingUtil#generateAllSaltingRanges}. Lower bound reads the first
+ // range, upper bound reads the last — so the scan displays as
+ // {@code [X'00', ...] - [X'03', ...]} for nBuckets=4. Reading the same entry
+ // for both bounds would display {@code [X'00', ...]} as both lower and upper,
+ // losing the span across salt buckets. Other prefix slots (viewIndexId, tenantId)
+ // have a single singleton range, so the choice doesn't matter there.
+ java.util.List<KeyRange> slot = ranges.get(d);
+ KeyRange lowerRange = slot.get(0);
+ KeyRange upperRange = slot.get(slot.size() - 1);
+ byte[] lb = lowerRange.getRange(KeyRange.Bound.LOWER);
+ byte[] ub = upperRange.getRange(KeyRange.Bound.UPPER);
+ // On a local-index scan, slot 0 holds the viewIndexId. Render it via V1's
+ // shifted-value format ({@code Short.MIN_VALUE + 1 → "1"}) so the explain-plan
+ // matches V1's output. Without this flag, the raw stored bytes display as
+ // {@code -32768}, diverging from V1 on every local-index explain assertion.
+ boolean changeViewIndexId = isLocalIndex && d == 0;
+ appendPKColumnValue(lower, context, table, lb, null, d, changeViewIndexId);
Review Comment:
This assumes every local-index scan has the viewIndexId in prefix slot 0, but salted local indexes prepend the salt byte before it. On those tables the formatter will interpret the salt bucket as a viewIndexId and leave the real viewIndexId unshifted, so the explain output is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3191291001
##########
phoenix-core/src/test/java/org/apache/phoenix/query/QueryPlanTest.java:
##########
@@ -213,6 +221,10 @@ public void testTenantSpecificConnWithLimit() throws Exception {
QueryUtil.getExplainPlan(rs));
}
+ // V2 limitation: DESC timestamp compound byte layout isn't yet decodable by the
+ // ExplainTable splitter (the schema iterator reports mismatched widths for
+ // DESC-inverted timestamp columns).
+ @org.junit.Ignore
Review Comment:
Is this breaking change with V2?
##########
phoenix-core/src/test/java/org/apache/phoenix/compile/WhereCompilerTest.java:
##########
@@ -427,16 +444,13 @@ public void testRowKeyFilter() throws SQLException {
Scan scan = plan.getContext().getScan();
Filter filter = scan.getFilter();
- assertEquals(
- new RowKeyComparisonFilter(
- constantComparison(CompareOperator.EQUAL,
- new SubstrFunction(Arrays.<Expression> asList(
- new RowKeyColumnExpression(ENTITY_ID,
- new RowKeyValueAccessor(ATABLE.getPKColumns(), 1)),
- LiteralExpression.newConstant(1), LiteralExpression.newConstant(3))),
- keyPrefix),
- QueryConstants.DEFAULT_COLUMN_FAMILY_BYTES),
- filter);
Review Comment:
If we can't assert this, is it not a regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "kadirozde (via GitHub)" <gi...@apache.org>.
kadirozde commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3192304544
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/scan/V2ExplainFormatter.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.scan;
+
+import java.text.Format;
+
+import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
+import org.apache.phoenix.compile.StatementContext;
+import org.apache.phoenix.compile.keyspace.KeySpace;
+import org.apache.phoenix.compile.keyspace.KeySpaceList;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.SortOrder;
+import org.apache.phoenix.schema.TableRef;
+import org.apache.phoenix.schema.types.PDataType;
+import org.apache.phoenix.util.StringUtil;
+
+/**
+ * V2-owned explain-plan keyRanges formatter.
+ * <p>
+ * Reads from {@link V2ScanArtifact#list()} directly rather than re-decoding the byte-
+ * encoded {@code ScanRanges} that drives actual scan execution. The artifact carries the
+ * pre-encoding, mathematical form of the scan, so inclusive-upper displays as
+ * {@code [*, 1]} (matching V1) instead of {@code [*, 2)} (what V2's compound byte
+ * emission produces after {@code nextKey(1) = 2}).
+ * <p>
+ * Current scope: single-space KeySpaceList with point or range ranges per dim. For
+ * multi-space lists the caller falls back to the legacy byte-decoding path in
+ * {@code ExplainTable.appendKeyRanges}, which produces different but semantically
+ * equivalent output. A future extension will handle the multi-space case by computing
+ * per-dim unions and emitting SkipScanFilter-style displays.
+ */
+public final class V2ExplainFormatter {
+
+ private V2ExplainFormatter() {
+ }
+
+ /**
+ * Build the keyRanges display string for a scan whose V2 artifact is {@code artifact}.
+ * Returns {@code null} if this formatter does not handle the input shape — the caller
+ * should fall back to the legacy formatter.
+ */
+ public static String appendKeyRanges(StatementContext context, TableRef tableRef,
+ V2ScanArtifact artifact) {
+ KeySpaceList list = artifact.list();
+ if (list.isUnsatisfiable()) {
+ return "";
+ }
+ PTable table = tableRef.getTable();
+ int nPk = artifact.nPkColumns();
+ int prefixSlots = artifact.prefixSlots();
+
+ // KeySpaceList.isEverything() means no user-dim constraint, but when prefix slots
+ // (salt / viewIndexId / tenantId) are present the scan is still narrowed to the
+ // tenant partition — render those prefix values. Without this, tenant-only queries
+ // display as empty brackets while V1 shows {@code ['tenantId']}.
+ if (list.isEverything()) {
+ return renderPrefixOnly(context, table, prefixSlots);
+ }
+ if (list.size() != 1) {
+ // Multi-space path not implemented yet; fall back.
+ return null;
+ }
+ KeySpace space = list.spaces().get(0);
+ boolean isLocalIndex = org.apache.phoenix.util.ScanUtil.isLocalIndex(context.getScan());
+
+ // KeySpace indexing is by absolute PK position ({@link KeySpaceExpressionVisitor}
+ // emits ranges via {@code KeySpace.single(pkPos, ...)} with {@code pkPos} being the
+ // column's absolute index in the PK). So {@code space.get(d)} — where d is an
+ // absolute PK index — returns the dim for PK column d. Prefix dims (viewIndexId /
+ // tenantId / salt) are always EVERYTHING inside the KeySpaceList because the visitor
+ // doesn't know about them; prefix values are rendered from scanRanges below.
+ //
+ // Fall back when a dim's raw bytes don't match the PK column's full fixed-width size.
+ // Happens when a scalar function (SUBSTR, FLOOR, ...) creates a KeyRange with truncated
+ // bytes (e.g. 3-byte substr prefix on an 8-byte LONG column). V1's ExplainTable reads
+ // post-processing ScanRanges bytes which have been zero-padded to the field width; V2's
+ // artifact carries the raw (pre-processing) bytes. Rather than duplicating V1's padding
+ // logic, defer to the legacy byte-decoding formatter.
+ for (int d = prefixSlots; d < nPk && d < space.nDims(); d++) {
+ KeyRange kr = space.get(d);
+ if (kr == KeyRange.EVERYTHING_RANGE || kr == KeyRange.IS_NULL_RANGE
+ || kr == KeyRange.IS_NOT_NULL_RANGE) {
+ continue;
+ }
+ PColumn col = table.getPKColumns().get(d);
+ PDataType type = col.getDataType();
+ if (!type.isFixedWidth()) {
+ continue;
+ }
+ Integer maxLen = col.getMaxLength();
+ Integer typeSize = type.getByteSize();
+ if (maxLen == null && typeSize == null) {
+ continue;
+ }
+ int expected = maxLen != null ? maxLen : typeSize;
+ byte[] lb = kr.getRange(KeyRange.Bound.LOWER);
+ byte[] ub = kr.getRange(KeyRange.Bound.UPPER);
+ if ((lb != null && lb.length != 0 && lb.length != expected)
+ || (ub != null && ub.length != 0 && ub.length != expected)) {
+ return null;
+ }
+ }
+
+ // Last-dim-to-display index: highest dim with a non-EVERYTHING range. V1 truncates
+ // at the first EVERYTHING past the prefix so the display doesn't end in `*,*,*`.
+ int lastConstrained = prefixSlots - 1;
+ for (int d = prefixSlots; d < nPk && d < space.nDims(); d++) {
+ KeyRange kr = space.get(d);
+ if (kr != KeyRange.EVERYTHING_RANGE) {
+ lastConstrained = d;
+ }
+ }
+ if (lastConstrained < prefixSlots) {
+ return "";
+ }
+
+ StringBuilder lower = new StringBuilder();
+ StringBuilder upper = new StringBuilder();
+ // Prefix columns (salt byte / viewIndexId / tenantId) aren't in the KeySpaceList —
+ // they're auto-populated slots. Read them from the ScanRanges's per-slot structure,
+ // which the caller already built. This keeps prefix display identical to V1's.
+ int prefixEmitted = 0;
+ if (context.getScanRanges() != null && !context.getScanRanges().getRanges().isEmpty()) {
+ java.util.List<java.util.List<KeyRange>> ranges = context.getScanRanges().getRanges();
+ for (int d = 0; d < prefixSlots && d < ranges.size(); d++) {
+ // Slot 0 on a salted table holds the full [0..nBuckets-1] salt-bucket range
+ // list after {@link ScanRanges}'s constructor populates it via
+ // {@link SaltingUtil#generateAllSaltingRanges}. Lower bound reads the first
+ // range, upper bound reads the last — so the scan displays as
+ // {@code [X'00', ...] - [X'03', ...]} for nBuckets=4. Reading the same entry
+ // for both bounds would display {@code [X'00', ...]} as both lower and upper,
+ // losing the span across salt buckets. Other prefix slots (viewIndexId, tenantId)
+ // have a single singleton range, so the choice doesn't matter there.
+ java.util.List<KeyRange> slot = ranges.get(d);
+ KeyRange lowerRange = slot.get(0);
+ KeyRange upperRange = slot.get(slot.size() - 1);
+ byte[] lb = lowerRange.getRange(KeyRange.Bound.LOWER);
+ byte[] ub = upperRange.getRange(KeyRange.Bound.UPPER);
+ // On a local-index scan, slot 0 holds the viewIndexId. Render it via V1's
+ // shifted-value format ({@code Short.MIN_VALUE + 1 → "1"}) so the explain-plan
+ // matches V1's output. Without this flag, the raw stored bytes display as
+ // {@code -32768}, diverging from V1 on every local-index explain assertion.
+ boolean changeViewIndexId = isLocalIndex && d == 0;
+ appendPKColumnValue(lower, context, table, lb, null, d, changeViewIndexId);
Review Comment:
Fixed it with commit e53169d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "apurtell (via GitHub)" <gi...@apache.org>.
apurtell commented on PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#issuecomment-5687773910
@kadirozde Would it be possible for you to go through the Copilot findings and click 'Resolve Conversation' button for everything you think does not apply?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PHOENIX-6791 WHERE optimizer redesign [phoenix]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2428:
URL: https://github.com/apache/phoenix/pull/2428#discussion_r3918225131
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/scan/V2ScanBuilder.java:
##########
@@ -0,0 +1,374 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace.scan;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.phoenix.compile.ScanRanges;
+import org.apache.phoenix.compile.keyspace.KeyRangeExtractor;
+import org.apache.phoenix.compile.keyspace.KeySpace;
+import org.apache.phoenix.compile.keyspace.KeySpaceList;
+import org.apache.phoenix.parse.HintNode.Hint;
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.query.QueryConstants;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.schema.SortOrder;
+import org.apache.phoenix.schema.types.PChar;
+
+import org.apache.phoenix.thirdparty.com.google.common.base.Optional;
+
+/**
+ * Scan-construction entry point for the V2 WHERE optimizer.
+ * <p>
+ * Pipeline:
+ *
+ * <pre>
+ * WhereOptimizerV2.run
+ * → ExpressionNormalizer → KeySpaceExpressionVisitor (produces KeySpaceList)
+ * → V2ScanBuilder.build (this class)
+ * → CompoundByteEncoderEmitter.overrideScanRows (in-envelope shapes)
+ * → context.setScanRanges / context.setV2ScanArtifact
+ * </pre>
+ *
+ * Dispatches on a classification of the {@link KeySpaceList} (see
+ * {@code docs/where-optimizer-v2-scan-construction.md} §"Classification tree"):
+ * <ul>
+ * <li>Class 1 DEGENERATE → {@link ScanRanges#NOTHING}</li>
+ * <li>Class 2 EVERYTHING → {@link ScanRanges#EVERYTHING}</li>
+ * <li>Class 3 POINT_LOOKUP_LIST → natively emitted via {@link CompoundByteEncoder}</li>
+ * <li>Classes 4a–4e (RANGE_SCAN subcases) and 5 (SKIP_SCAN_LIST) → route through the
+ * {@link KeyRangeExtractor} adapter to produce V1-shaped CNF; scan start/stop bytes
+ * are then sourced from {@link CompoundByteEncoder} (via
+ * {@link CompoundByteEncoderEmitter} in {@code WhereOptimizerV2.run}) for shapes in
+ * the encoder's proven envelope.</li>
+ * </ul>
+ * Downstream consumers (SkipScanFilter, ScanRanges.isPointLookup, explain-plan
+ * formatter, local-index pruning) read from the ScanRanges this builder produces.
+ * V2-owned metadata is attached via {@link V2ScanArtifact} so the explain-plan formatter
+ * renders from the pre-encoding {@link KeySpaceList} rather than the post-encoding bytes.
+ */
+public final class V2ScanBuilder {
+
+ private V2ScanBuilder() {
+ }
+
+ /**
+ * Inputs gathered at {@code WhereOptimizerV2.run} and passed to the scan builder.
+ * All fields are read-only.
+ */
+ public static final class Inputs {
+ public final KeySpaceList list;
+ public final PTable table;
+ public final RowKeySchema schema;
+ public final int nPkColumns;
+ public final int prefixSlots;
+ public final Integer nBuckets;
+ public final boolean isSalted;
+ public final boolean isMultiTenant;
+ public final boolean isSharedIndex;
+ public final byte[] tenantIdBytes;
+ public final Set<Hint> hints;
+ public final int cartesianBound;
+ public final Optional<byte[]> minOffset;
+
+ public Inputs(KeySpaceList list, PTable table, RowKeySchema schema, int nPkColumns,
+ int prefixSlots, Integer nBuckets, boolean isSalted, boolean isMultiTenant,
+ boolean isSharedIndex, byte[] tenantIdBytes, Set<Hint> hints, int cartesianBound,
+ Optional<byte[]> minOffset) {
+ this.list = list;
+ this.table = table;
+ this.schema = schema;
+ this.nPkColumns = nPkColumns;
+ this.prefixSlots = prefixSlots;
+ this.nBuckets = nBuckets;
+ this.isSalted = isSalted;
+ this.isMultiTenant = isMultiTenant;
+ this.isSharedIndex = isSharedIndex;
+ this.tenantIdBytes = tenantIdBytes;
+ this.hints = hints;
+ this.cartesianBound = cartesianBound;
+ this.minOffset = minOffset;
+ }
+ }
+
+ /**
+ * Output of the scan builder. For now this is a thin wrapper around
+ * {@link ScanRanges} (the existing type), leaving room to grow into a richer V2-owned
+ * adapter as more responsibilities move into this class.
+ */
+ public static final class Result {
+ public final ScanRanges scanRanges;
+ /**
+ * {@code true} iff the builder's classification of the emitted key space is "matches
+ * nothing" — the caller short-circuits the residual and returns {@code null}. Distinct
+ * from {@code scanRanges.isDegenerate()} only insofar as it's set by the builder's
+ * own classification path (not always derivable from {@code scanRanges}).
+ */
+ public final boolean isNothing;
+
+ public Result(ScanRanges scanRanges, boolean isNothing) {
+ this.scanRanges = scanRanges;
+ this.isNothing = isNothing;
+ }
+
+ public static Result nothing() {
+ return new Result(ScanRanges.NOTHING, true);
+ }
+
+ public static Result everything() {
+ return new Result(ScanRanges.EVERYTHING, false);
+ }
+ }
+
+ /**
+ * Build a {@link ScanRanges} from the given {@link KeySpaceList} and context.
+ * <p>
+ * Follows the classification tree in {@code docs/where-optimizer-v2-scan-construction.md}
+ * §"Classification tree". Shapes with a native V2 emission path are handled directly;
+ * shapes routed through the {@link KeyRangeExtractor} adapter produce the V1-projected
+ * per-slot CNF shape that {@link org.apache.phoenix.compile.ScanRanges#create} + the
+ * downstream {@code ScanUtil.setKey} consume.
+ * <p>
+ * Currently native classes:
+ * <ul>
+ * <li><b>1 DEGENERATE</b> — {@code list.isUnsatisfiable()} → {@link ScanRanges#NOTHING}.</li>
+ * <li><b>2 EVERYTHING</b> — {@code list.isEverything() && !prefixSlots && !minOffset}
+ * → {@link ScanRanges#EVERYTHING}.</li>
+ * <li><b>3 POINT_LOOKUP_LIST</b> — every space all-single-key across every productive
+ * dim past prefix, {@code list.size() ≥ 2} (single-space single-tuple routes through
+ * adapter to preserve DESC var-width byte shape). Emitted directly via
+ * {@link CompoundByteEncoder}, preserving cross-dim tuple correlation.</li>
+ * </ul>
+ * Classes 4 (RANGE_SCAN subcases) and 5 (SKIP_SCAN_LIST) currently route through the
+ * {@link KeyRangeExtractor} adapter to produce the V1-shaped CNF that
+ * {@link SkipScanFilter} consumes. {@link CompoundByteEncoderEmitter} then overrides
+ * {@code scan.startRow}/{@code stopRow} with encoder-sourced bytes for in-envelope
+ * shapes (see {@code docs/where-optimizer-v2-scan-construction.md} §"Byte emission
+ * envelope"). Native emission for classes 4 and 5 is PHOENIX-6791 follow-up work.
+ */
+ public static Result build(Inputs in) {
+ // Class 1: DEGENERATE.
+ if (in.list.isUnsatisfiable()) {
+ return Result.nothing();
+ }
+ // Class 2: EVERYTHING.
+ if (in.list.isEverything()) {
+ if (in.prefixSlots == 0 && !in.minOffset.isPresent()) {
+ return Result.everything();
+ }
+ }
+
+ // Class 3: POINT_LOOKUP_LIST.
+ if (isPointLookupList(in)) {
+ Result pl = buildPointLookupList(in);
+ if (pl != null) {
+ return pl;
+ }
+ // Native path opted out (encoder refused a space, e.g., IS_NULL sentinel).
+ // Fall through to the classical adapter.
+ }
+
+ // Classes 4 (RANGE_SCAN subcases) and 5 (SKIP_SCAN_LIST): adapter.
+
+ KeyRangeExtractor.Result extract = KeyRangeExtractor.extract(
+ in.list, in.nPkColumns, in.cartesianBound, in.prefixSlots, in.schema);
+ if (extract.isNothing()) {
+ return Result.nothing();
+ }
+
+ // Build CNF exactly the way WhereOptimizerV2.run does today: prefix slots (salt /
+ // viewIndexId / tenantId) + extractor-emitted user tail.
+ List<List<KeyRange>> cnf = new ArrayList<>(in.nPkColumns);
+ if (in.isSalted) {
+ // Salt byte placeholder. ScanRanges.isPointLookup requires a singleton point range
+ // (not EVERYTHING) for the whole query to classify as a point lookup when the user
+ // slots also carry single keys.
+ cnf.add(Collections.singletonList(
+ PChar.INSTANCE.getKeyRange(QueryConstants.SEPARATOR_BYTE_ARRAY, SortOrder.ASC)));
+ }
+ if (in.isSharedIndex) {
+ byte[] viewIndexBytes = in.table.getviewIndexIdType().toBytes(in.table.getViewIndexId());
+ cnf.add(Collections.singletonList(KeyRange.getKeyRange(viewIndexBytes)));
+ }
+ if (in.isMultiTenant) {
+ cnf.add(Collections.singletonList(KeyRange.getKeyRange(in.tenantIdBytes)));
+ }
+ boolean useSkipScan = extract.useSkipScan;
+ if (in.hints != null) {
+ if (in.hints.contains(Hint.SKIP_SCAN)) {
+ useSkipScan = true;
+ } else if (in.hints.contains(Hint.RANGE_SCAN)) {
+ useSkipScan = false;
+ }
+ }
+ for (int i = 0; i < extract.ranges.size(); i++) {
+ cnf.add(extract.ranges.get(i));
+ }
+ int[] slotSpan = new int[cnf.size()];
+ if (extract.slotSpan.length > 0) {
+ int len = Math.min(extract.slotSpan.length, slotSpan.length - in.prefixSlots);
+ if (len > 0) {
+ System.arraycopy(extract.slotSpan, 0, slotSpan, in.prefixSlots, len);
+ }
+ }
+
+ ScanRanges scanRanges = ScanRanges.create(in.schema, cnf, slotSpan, in.nBuckets, useSkipScan,
+ in.table.getRowTimestampColPos(), in.minOffset);
+ return new Result(scanRanges, false);
+ }
+
+ /**
+ * Classifier: is every space in the list all-single-key across every productive dim,
+ * with no IS_NULL / IS_NOT_NULL sentinels? This is the RVC-IN / RVC-equality OR shape.
+ * <p>
+ * Restricted to multi-space lists (size ≥ 2). Single-space all-pinned shapes flow
+ * through the classical path which is already byte-identical to V1 (proven by parity
+ * harness across 142 tests); routing them through the native path would change byte
+ * output unnecessarily and break byte-shape assertions on point lookups.
+ */
+ private static boolean isPointLookupList(Inputs in) {
+ if (in.list.isUnsatisfiable() || in.list.isEverything()) {
+ return false;
+ }
+ if (in.list.size() < 2) {
+ return false;
+ }
+ if (in.isSalted) {
+ // Salted tables: each row's salt byte is hash(row_key_no_salt) % nBuckets; the
+ // native path can't replicate that hashing here. ScanRanges.create does it
+ // correctly for point-lookup shapes via getPointKeys; defer to the adapter.
+ return false;
+ }
+ if (in.minOffset.isPresent()) {
+ // RVC-OFFSET uses getScanRange().getLowerRange() downstream; the classical path's
+ // byte layout is what that consumer expects. Stay on the adapter.
+ return false;
+ }
+ // Every space must be all-single-key past prefix, every dim must be constrained
+ // (no middle gaps), and no IS_NULL / IS_NOT_NULL sentinels.
+ int nPk = in.nPkColumns;
+ int productiveDims = 0;
+ for (KeySpace s : in.list.spaces()) {
+ int thisProductive = 0;
+ for (int d = in.prefixSlots; d < nPk; d++) {
+ KeyRange r = s.get(d);
+ if (r == KeyRange.EVERYTHING_RANGE) {
+ if (thisProductive > 0) return false; // middle gap
+ continue;
+ }
Review Comment:
The point-lookup classifier still accepts leading `EVERYTHING` dimensions. For a variable-width leading PK, `CompoundByteEncoder.encodeLower` emits a separator for that wildcard, and `buildPointLookupList` then treats the resulting bytes as an exact point key. A query such as an RVC `IN` over PK2/PK3 with unconstrained VARCHAR PK1 can therefore scan only the synthetic empty-PK1 keys and miss valid rows. Reject any unconstrained user dimension as the method contract requires, and let the adapter handle partial keys.
##########
phoenix-core/src/test/java/org/apache/phoenix/query/BaseTest.java:
##########
@@ -194,6 +194,32 @@
public abstract class BaseTest {
public static final String DRIVER_CLASS_NAME_ATTRIB = "phoenix.driver.class.name";
protected static final String NULL_STRING = "NULL";
+
+ /**
+ * Captures the {@code -Dphoenix.where.optimizer.v2.enabled=...} value (if any) layered
+ * into the test driver via {@link #initDriver}. Tests use {@link #isV2Optimizer()} to
+ * branch their expected output between V1 and V2 forms when the optimizer's compound
+ * scan emission produces a tighter scan range or different residual filter shape.
+ */
+ private static volatile String v2OptimizerEnabledOverride;
+
+ /**
+ * True when the V2 WHERE optimizer is enabled for this test JVM (either via the
+ * {@code -Dphoenix.where.optimizer.v2.enabled=true} JVM arg or the codebase default
+ * if it has been flipped on). Returns false when the property is unset and the
+ * codebase default is V1. Use this from IT/UT tests whose expected scan-range or
+ * EXPLAIN output differs between V1 and V2.
Review Comment:
The documented fallback is stale: the current codebase default is V2 (`DEFAULT_WHERE_OPTIMIZER_V2_ENABLED` is `true`), not V1. Describe the precedence without claiming an incorrect default so tests do not select the wrong expected plan.
##########
pom.xml:
##########
@@ -177,6 +177,12 @@
<numForkedIT>7</numForkedIT>
<it.failIfNoSpecifiedTests>false</it.failIfNoSpecifiedTests>
<surefire.failIfNoSpecifiedTests>false</surefire.failIfNoSpecifiedTests>
+ <!-- WHERE optimizer V2 flag, forwarded into forked test JVMs by the surefire and
+ failsafe configurations below. Default empty so the connectionless test driver
+ picks the codebase default (V1). Override with
+ `mvn test -Dphoenix.where.optimizer.v2.enabled=true` to run the same suites
+ under V2. -->
Review Comment:
This describes the default backwards: `DEFAULT_WHERE_OPTIMIZER_V2_ENABLED` is `true`, so an empty property runs V2 and passing `true` does not switch modes. This can mislead developers about which optimizer the default test run covers; document V2 as the default and use `false` for a V1 run.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/query/KeyRange.java:
##########
@@ -383,14 +383,24 @@ public boolean lowerUnbound() {
return lowerRange == UNBOUND;
}
+ // Cached hash. 0 means uncomputed (the chance of a legitimate 0 hash is negligible and
+ // even if it hits we just recompute). KeyRange is effectively immutable once constructed
+ // (fields are final), so memoization is safe.
+ private int cachedHashCode;
+
@Override
public int hashCode() {
+ int h = cachedHashCode;
+ if (h != 0) {
+ return h;
+ }
final int prime = 31;
int result = 1;
result = prime * result + Arrays.hashCode(lowerRange);
if (lowerRange != null) result = prime * result + (lowerInclusive ? 1231 : 1237);
result = prime * result + Arrays.hashCode(upperRange);
if (upperRange != null) result = prime * result + (upperInclusive ? 1231 : 1237);
+ cachedHashCode = result;
return result;
}
Review Comment:
`KeyRange` is not immutable: its bound fields are protected and `readFields` replaces them. If `hashCode()` is called before a Writable instance is reused for deserialization, this cache retains the old value, violating the `equals`/`hashCode` contract and breaking hash-based collections. Remove the memoization (or comprehensively invalidate it on every mutation, including subclass mutations).
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/WhereOptimizerV2.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace;
+
+import java.sql.SQLException;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.phoenix.compile.ScanRanges;
+import org.apache.phoenix.compile.StatementContext;
+import org.apache.phoenix.compile.WhereOptimizer;
+import org.apache.phoenix.compile.keyspace.scan.V2ScanBuilder;
+import org.apache.phoenix.expression.Expression;
+import org.apache.phoenix.expression.LiteralExpression;
+import org.apache.phoenix.parse.HintNode.Hint;
+import org.apache.phoenix.query.QueryServices;
+import org.apache.phoenix.query.QueryServicesOptions;
+import org.apache.phoenix.schema.PName;
+import org.apache.phoenix.schema.PColumn;
+import org.apache.phoenix.schema.PTable;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.util.ScanUtil;
+
+import org.apache.phoenix.thirdparty.com.google.common.base.Optional;
+
+/**
+ * Entry point for the N-dimensional key-space WHERE optimizer. Pipes an expression through
+ * {@link ExpressionNormalizer}, {@link KeySpaceExpressionVisitor}, {@link KeyRangeExtractor},
+ * and finally {@link ScanRanges#create}, then strips fully-consumed nodes via
+ * {@link WhereOptimizer.RemoveExtractedNodesVisitor}.
+ * <p>
+ * The driver is invoked in place of the legacy {@link WhereOptimizer} visitor when the
+ * {@link QueryServices#WHERE_OPTIMIZER_V2_ENABLED} flag is set. Both the legacy path and this
+ * one write the same shape to {@code context.setScanRanges(...)} and return an Expression
+ * representing the residual filter.
+ */
+public final class WhereOptimizerV2 {
+
+ private WhereOptimizerV2() {
+ }
+
+ public static Expression run(StatementContext context, Set<Hint> hints, Expression whereClause,
+ Set<Expression> extractNodes, Optional<byte[]> minOffset) throws SQLException {
+
+ PTable table = context.getCurrentTable().getTable();
+ RowKeySchema schema = table.getRowKeySchema();
+ Integer nBuckets = table.getBucketNum();
+ boolean isSalted = nBuckets != null;
+ PName tenantId = context.getConnection().getTenantId();
+ boolean isMultiTenant = tenantId != null && table.isMultiTenant();
+ boolean isSharedIndex = table.getViewIndexId() != null;
+ byte[] tenantIdBytes = isMultiTenant
+ ? ScanUtil.getTenantIdBytes(schema, isSalted, tenantId, isSharedIndex)
+ : null;
+
+ // Short-circuits matching WhereOptimizer.pushKeyExpressionsToScan.
+ if (whereClause == null && !isMultiTenant && !isSharedIndex && !minOffset.isPresent()) {
+ context.setScanRanges(ScanRanges.EVERYTHING);
+ return whereClause;
+ }
+ if (LiteralExpression.isBooleanFalseOrNull(whereClause)) {
+ context.setScanRanges(ScanRanges.NOTHING);
+ return null;
+ }
+
+ // FROM-less SELECT (e.g., `SELECT 1` or expression-only queries) resolves to a
+ // synthetic PTable with no PK columns. Phoenix represents this by returning null
+ // from getPKColumns(). There's nothing the optimizer can narrow in that case — leave
+ // the scan as EVERYTHING and return the residual expression unchanged so the
+ // executor evaluates it at scan time.
+ List<PColumn> pkColumns = table.getPKColumns();
+ if (pkColumns == null || pkColumns.isEmpty()) {
+ context.setScanRanges(ScanRanges.EVERYTHING);
+ return whereClause;
+ }
+ int nPk = pkColumns.size();
+ int prefixSlots = (isSalted ? 1 : 0) + (isSharedIndex ? 1 : 0) + (isMultiTenant ? 1 : 0);
+
+ // Step 1: normalize + visit. The normalized tree is what the residual filter is built
+ // from — extracted Expression nodes come from the normalized tree, not the caller's
+ // original tree, so applying {@link WhereOptimizer.RemoveExtractedNodesVisitor} against
+ // the original would find nothing to strip for RVC-inequality and IN rewrites.
+ KeySpaceList keySpaceList;
+ Set<Expression> consumed = (extractNodes == null) ? new HashSet<Expression>() : extractNodes;
+ Expression residualInput = whereClause;
+ Set<Expression> visitorConsumed = Collections.emptySet();
+ if (whereClause == null) {
+ keySpaceList = KeySpaceList.everything(nPk);
+ } else {
+ Expression normalized = ExpressionNormalizer.normalize(whereClause);
+ residualInput = normalized;
+ KeySpaceExpressionVisitor visitor = new KeySpaceExpressionVisitor(table);
+ KeySpaceExpressionVisitor.Result r = normalized.accept(visitor);
+ if (r == null || r.list().isEverything()) {
+ keySpaceList = KeySpaceList.everything(nPk);
+ } else if (r.list().isUnsatisfiable()) {
+ // PHOENIX-6669 short-circuit: degeneracy detected uniformly across all PK positions.
+ context.setScanRanges(ScanRanges.NOTHING);
+ return null;
+ } else {
+ keySpaceList = r.list();
+ visitorConsumed = r.consumed();
+ }
+ }
+
+ int bound = context.getConnection().getQueryServices().getConfiguration()
+ .getInt(QueryServices.WHERE_OPTIMIZER_V2_CARTESIAN_BOUND,
+ QueryServicesOptions.DEFAULT_WHERE_OPTIMIZER_V2_CARTESIAN_BOUND);
+
+ V2ScanBuilder.Inputs inputs = new V2ScanBuilder.Inputs(keySpaceList, table, schema, nPk,
+ prefixSlots, nBuckets, isSalted, isMultiTenant, isSharedIndex, tenantIdBytes, hints,
+ bound, minOffset);
+ V2ScanBuilder.Result r = V2ScanBuilder.build(inputs);
+ if (r.isNothing) {
+ context.setScanRanges(ScanRanges.NOTHING);
+ return null;
+ }
+ boolean emittedEverything = r.scanRanges == ScanRanges.EVERYTHING;
+ context.setScanRanges(r.scanRanges);
+ // Attach the V2 artifact so downstream consumers (explain-plan formatter) can read
+ // the logical KeySpaceList rather than the byte-encoded ScanRanges. Skipped for
+ // EVERYTHING since there's nothing to display anyway.
+ if (!emittedEverything) {
+ context.setV2ScanArtifact(new org.apache.phoenix.compile.keyspace.scan.V2ScanArtifact(
+ keySpaceList, nPk, prefixSlots));
+ }
+ // Override scan start/stop rows with CompoundByteEncoder output for shapes in the
+ // encoder's proven envelope. The encoder's bytes preserve trailing separators that
+ // ScanUtil.setKey's tail-strip would drop, and its multi-space list envelope preserves
+ // cross-dim tuple correlation that per-slot projection loses — see
+ // docs/where-optimizer-v2-scan-construction.md. RVC OFFSET is skipped because
+ // RVCOffsetCompiler reads scan.startRow to build the paging cursor and is sensitive
+ // to the classical path's exact byte layout. See QueryMoreIT.testRVCOnDescWithLeadingPKEquality.
+ if (!emittedEverything && !minOffset.isPresent()
+ && org.apache.phoenix.compile.keyspace.scan.CompoundByteEncoderEmitter.isInScope(
+ keySpaceList, schema, prefixSlots, isSalted)) {
+ org.apache.phoenix.compile.keyspace.scan.CompoundByteEncoderEmitter.overrideScanRows(
+ context.getScan(), keySpaceList, schema, prefixSlots,
+ buildPrefixBytes(isSalted, isSharedIndex, isMultiTenant, table, tenantIdBytes));
+ }
+
+ // If the emitted scan range is "everything" (no leading-PK narrowing survived the
+ // extract pass, e.g. a predicate on a non-leading PK column with no leading
+ // constraint), the visitor may still have populated consumed nodes for those
+ // predicates, but since they didn't influence the scan range, the residual filter
+ // must retain them for correctness. Match v1 semantics by clearing consumed in that
+ // case.
+ //
+ // Additionally, when the leading productive slot (the first user-PK dim at
+ // {@code prefixSlots}) is EVERYTHING in every space but a trailing slot is
+ // constrained, we cannot safely install a SkipScanFilter. Its setNextCellHint path
+ // builds startKey by concatenating each slot's lower bound — an EVERYTHING slot
+ // contributes zero bytes, producing a seek hint shorter than (and lex-less than)
+ // any row already past the trailing-slot boundary. HBase rejects the backward
+ // seek with {@code "next hint must come after previous hint"}. Prefix bytes (salt
+ // bucket, viewIndexId, tenantId) from {@code [0, prefixSlots)} are concrete, but
+ // they only anchor the scan's start/stop — the hint construction still uses the
+ // user slots and the same invariant applies. The predicate must stay in the
+ // residual BooleanExpressionFilter.
+ // Exception: when the caller forces SKIP_SCAN via hint, honor the hint —
+ // V2ScanBuilder installs the SkipScanFilter anyway and the caller accepts the
+ // associated risk (V1 has the same behavior); consume so the shape matches V1's
+ // compile-plan output.
+ boolean forcedSkipScan = hints != null && hints.contains(Hint.SKIP_SCAN);
+ boolean leadingEverythingPastPrefix = !emittedEverything && !forcedSkipScan
+ && hasLeadingEverythingAt(keySpaceList, prefixSlots);
+ if (!emittedEverything && !leadingEverythingPastPrefix) {
+ consumed.addAll(visitorConsumed);
Review Comment:
A non-`EVERYTHING` scan does not prove that every visitor-consumed predicate survived later approximation. `KeySpaceList.widenToBudget` can replace constrained dimensions with `EVERYTHING`, and `emitV1Projection` can drop trailing slots, yet this adds all original predicates to the removal set. For an over-budget conjunction, a dropped predicate is then absent from both the scan and residual filter, admitting rows that do not satisfy the WHERE clause. Propagate exactness/dropped-dimension metadata from the algebra and scan builder and retain the affected expressions in the residual.
##########
phoenix-core-client/src/main/java/org/apache/phoenix/compile/keyspace/KeyRangeExtractor.java:
##########
@@ -0,0 +1,1190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.phoenix.compile.keyspace;
+
+import java.math.BigInteger;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.phoenix.query.KeyRange;
+import org.apache.phoenix.schema.RowKeySchema;
+import org.apache.phoenix.util.ScanUtil;
+
+/**
+ * Converts a final {@link KeySpaceList} into the shape
+ * {@link org.apache.phoenix.compile.ScanRanges#create} consumes:
+ * {@code List<List<KeyRange>> ranges}, {@code int[] slotSpan}, {@code boolean useSkipScan}.
+ * <p>
+ * <b>The V1 projection (default output shape).</b> The legacy optimizer produced one
+ * {@link KeyRange} list per PK column ("slot"): the disjunction of every narrowing the
+ * WHERE clause places on that column. {@link org.apache.phoenix.compile.ScanRanges}
+ * and {@link org.apache.phoenix.filter.SkipScanFilter} are built against that shape.
+ * V2 computes its narrowing as a {@link KeySpaceList} (disjunction of N-dim boxes)
+ * and, by projecting each {@link KeySpace} onto each PK column and coalescing per column,
+ * produces the same V1-compatible shape. This is the role of
+ * {@link #emitV1Projection}. The method name reflects its job: it's the boundary layer
+ * where V2's N-dim key-space algebra is converted into V1's per-slot disjunctions so
+ * the existing downstream machinery can consume it unchanged.
+ * <p>
+ * <b>Compound emission (optional optimization).</b> For some shapes a tighter scan is
+ * possible by concatenating per-dim bytes into a single compound {@link KeyRange} per
+ * {@link KeySpace} — preserving cross-dim tuple correlation at the byte level. Compound
+ * emission uses one output slot with {@code slotSpan = maxProductiveLen - 1}; the start
+ * and stop rows then narrow to the exact compound interval (e.g. a 15-tuple RVC-IN
+ * becomes a POINT LOOKUP on 15 compound keys rather than a SkipScan over 15 per-column
+ * disjunctions). When compound emission is unsafe — single productive dim, IS_NULL
+ * sentinels, middle-EVERYTHING gap, mixed-width coalesced ranges with non-point values
+ * — the extractor falls back to {@link #emitV1Projection}.
+ * <p>
+ * Once the legacy V1 optimizer is removed and downstream utilities ({@code ScanUtil.setKey},
+ * {@code ScanRanges.create}'s special cases for {@code IS_NULL_RANGE}, etc.) are
+ * simplified to match the compound shape natively, the per-slot fallback can be
+ * deleted and compound emission becomes the sole path.
+ * <p>
+ * Correctness guarantee (for both paths): for every row the original predicate matches,
+ * the emitted scan contains it. False positives (rows in the emitted scan that don't
+ * satisfy the predicate) are handled by the residual filter. The cartesian-bound
+ * widening rule (drop trailing dims when the list size would exceed a threshold) is
+ * applied inside the extractor for the per-slot path and upstream in
+ * {@link KeySpaceList} for the compound path.
+ * <p>
+ * Prefix slots (salt byte, view-index id, tenant id) are prepended by {@link WhereOptimizerV2}
+ * at CNF-build time; this class emits only the user tail.
+ */
+public final class KeyRangeExtractor {
+
+ /** Result of an extraction pass, shaped exactly like the inputs to {@code ScanRanges.create}. */
+ public static final class Result {
+ public final List<List<KeyRange>> ranges;
+ public final int[] slotSpan;
+ public final boolean useSkipScan;
+
+ public Result(List<List<KeyRange>> ranges, int[] slotSpan, boolean useSkipScan) {
+ this.ranges = ranges;
+ this.slotSpan = slotSpan;
+ this.useSkipScan = useSkipScan;
+ }
+
+ public boolean isNothing() {
+ return ranges.size() == 1 && ranges.get(0).size() == 1
+ && ranges.get(0).get(0) == KeyRange.EMPTY_RANGE;
+ }
+
+ public boolean isEverything() {
+ return ranges.isEmpty();
+ }
+ }
+
+ public static Result everything() {
+ return new Result(Collections.<List<KeyRange>>emptyList(), new int[0], false);
+ }
+
+ public static Result nothing() {
+ return new Result(
+ Collections.<List<KeyRange>>singletonList(Collections.singletonList(KeyRange.EMPTY_RANGE)),
+ ScanUtil.SINGLE_COLUMN_SLOT_SPAN, false);
+ }
+
+ private KeyRangeExtractor() {
+ }
+
+ /**
+ * Legacy entry point used by tests that don't have a schema handy. Emits per-slot output
+ * (pre-compound-emission behavior). Kept for test compatibility.
+ */
+ public static Result extract(KeySpaceList list, int nPkColumns, int cartesianBound) {
+ return emitV1ProjectionStopAtGap(list, nPkColumns, cartesianBound, 0);
+ }
+
+ /**
+ * Legacy entry point for schema-less tests with prefix slots.
+ */
+ public static Result extract(KeySpaceList list, int nPkColumns, int cartesianBound,
+ int prefixSlots) {
+ return emitV1ProjectionStopAtGap(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+
+ /**
+ * Compound-emission entry point: emits one compound {@link KeyRange} per {@link KeySpace}
+ * in the list, into a single output slot with {@code slotSpan = maxProductiveLen - 1}.
+ * Requires a schema to concatenate per-dim bytes with correct separator handling.
+ */
+ public static Result extract(KeySpaceList list, int nPkColumns, int cartesianBound,
+ int prefixSlots, RowKeySchema schema) {
+ if (schema == null) {
+ return emitV1ProjectionStopAtGap(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+ if (list.isUnsatisfiable()) {
+ return nothing();
+ }
+ if (list.isEverything()) {
+ return everything();
+ }
+
+ // Scan spaces to find the widest productive extent and whether every space has a
+ // middle-EVERYTHING gap. When every space has a middle gap past the prefix, emit
+ // per-slot so SkipScanFilter can narrow the trailing dim independently. When only
+ // some spaces have middle gaps, compound-emit each space independently so the
+ // non-middle-gap spaces can anchor a tight compound startRow.
+ int minProductiveStart = nPkColumns;
+ int maxProductiveEnd = prefixSlots;
+ boolean allSpacesHaveMiddleGap = true;
+ for (KeySpace ks : list.spaces()) {
+ int start = firstConstrainedDim(ks, prefixSlots);
+ if (start < 0) {
+ // Space is EVERYTHING past the prefix — whole list is EVERYTHING from our view.
+ return everything();
+ }
+ if (start < minProductiveStart) minProductiveStart = start;
+ int endStrict = firstProductiveStopStrict(ks, prefixSlots);
+ int endAny = firstProductiveStopAnyPrefix(ks, prefixSlots);
+ if (endAny > maxProductiveEnd) maxProductiveEnd = endAny;
+ // Detect a middle gap by comparing the strict stop (first EVERYTHING past prefix)
+ // against the any-prefix stop (last constrained dim past prefix). They diverge
+ // iff there's an EVERYTHING dim BEFORE the last constrained dim, i.e. a gap.
+ boolean hasMiddleGap = start == prefixSlots && endStrict < endAny;
+ if (!hasMiddleGap) {
+ allSpacesHaveMiddleGap = false;
+ }
+ }
+
+ // Leading EVERYTHING past the prefix, or EVERY space has a middle gap: emit per-slot.
+ // The per-slot SkipScanFilter handles narrowing past the gap and ScanRanges reports
+ // boundPkColumnCount correctly for the local-index-pruning heuristic.
+ if (minProductiveStart > prefixSlots || allSpacesHaveMiddleGap) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+
+
+ // Single-space, single-productive-dim: trivial case with no compound benefit.
+ // Using compound emission would pre-build bytes with separators, then ScanRanges.create
+ // (via getPointKeys -> ScanUtil.setKey) re-appends separator bytes for DESC fields,
+ // producing wider-than-correct scan. Per-slot emission lets ScanRanges process the
+ // range once with the real schema, matching V1's byte output exactly.
+ if (list.spaces().size() == 1 && (maxProductiveEnd - prefixSlots) == 1) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+
+ // If any space has IS_NULL_RANGE / IS_NOT_NULL_RANGE at ANY productive dim, route
+ // to per-slot emission so ScanRanges receives the IS_NULL / IS_NOT_NULL sentinel
+ // intact. ScanRanges.create has special-case handling for IS_NULL_RANGE (producing
+ // the correct empty-bytes + separator boundary). Compound emission would collapse
+ // the sentinel into either a degenerate half-open range or a zero-length
+ // single-key at the wrong byte position, both of which produce scan rows that
+ // skip actual null-value rows.
+ for (KeySpace ks : list.spaces()) {
+ for (int d = prefixSlots; d < ks.nDims(); d++) {
+ KeyRange dim = ks.get(d);
+ if (dim == KeyRange.IS_NULL_RANGE || dim == KeyRange.IS_NOT_NULL_RANGE) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+ }
+ }
+
+ // Mixed comparator safety gate: SkipScanFilter uses a single BytesComparator per
+ // compound slot, derived from schema.getField(rowKeyPosition) — i.e. the slot's
+ // LEADING field. When the compound spans multiple fields that require different
+ // comparators (e.g. ASC fixed-width BIGINT leading + DESC variable-width DECIMAL
+ // trailing), the leading-field comparator produces wrong results for the trailing
+ // field's bytes. Concretely: DESC var-width uses DescVarLengthFastByteComparisons
+ // which handles the variable-length DESC sort correctly; plain lex comparison
+ // (ASC fixed) does not. Fall back to per-slot emission so each slot gets its own
+ // comparator. See SortOrderIT.testSkipScanCompare.
+ if (prefixSlots < maxProductiveEnd) {
+ org.apache.phoenix.schema.ValueSchema.Field leadingField = schema.getField(prefixSlots);
+ org.apache.phoenix.util.ScanUtil.BytesComparator leadingCmp =
+ org.apache.phoenix.util.ScanUtil.getComparator(leadingField);
+ for (int d = prefixSlots + 1; d < maxProductiveEnd; d++) {
+ org.apache.phoenix.schema.ValueSchema.Field f = schema.getField(d);
+ if (org.apache.phoenix.util.ScanUtil.getComparator(f) != leadingCmp) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+ }
+ }
+
+ // (Compound-too-wide safety gate moved below, after compound window is computed.)
+
+ // IN-list-on-leading + range-on-trailing gate: when the leading productive dim is a
+ // single-key per space but points differ across spaces (classical IN-list shape)
+ // AND some later dim has a non-single-key range, route to per-slot emission. V2's
+ // compound-per-space form collapses the two-dimensional shape `[IN-list] × [range]`
+ // into one compound slot with {@code slotSpan = compoundLen - 1}, which under-
+ // counts PK columns covered (trailing unconstrained cols aren't accounted for).
+ // Server-side consumers that walk scans per-slot (e.g. {@link org.apache.phoenix
+ // .coprocessor.UncoveredGlobalIndexRegionScanner} /
+ // {@link org.apache.phoenix.coprocessor.CDCGlobalIndexRegionScanner} decode
+ // index-row keys to reconstruct data-row keys; ordering depends on how
+ // {@link org.apache.phoenix.filter.SkipScanFilter#intersect} steps the schema
+ // cursor, which reads {@code slotSpan[0]}) rely on the per-slot shape. Per-slot
+ // emission produces `[IN-list]` on slot 0 and `[range]` on slot 1 with the correct
+ // trailing-col coverage in slotSpan. See CDCQueryIT.testSelectCDC with salted
+ // data tables.
+ if (prefixSlots < maxProductiveEnd && list.spaces().size() >= 2) {
+ boolean leadingIsInList = true;
+ KeyRange firstLeading = null;
+ boolean leadingDiffers = false;
+ for (KeySpace ks : list.spaces()) {
+ KeyRange leading = ks.get(prefixSlots);
+ if (!leading.isSingleKey() || leading == KeyRange.IS_NULL_RANGE
+ || leading == KeyRange.IS_NOT_NULL_RANGE) {
+ leadingIsInList = false;
+ break;
+ }
+ if (firstLeading == null) {
+ firstLeading = leading;
+ } else if (!firstLeading.equals(leading)) {
+ leadingDiffers = true;
+ }
+ }
+ if (leadingIsInList && leadingDiffers) {
+ boolean laterRangeExists = false;
+ outer:
+ for (KeySpace ks : list.spaces()) {
+ for (int d = prefixSlots + 1; d < maxProductiveEnd; d++) {
+ KeyRange dim = ks.get(d);
+ if (dim == KeyRange.EVERYTHING_RANGE) continue;
+ if (!dim.isSingleKey()) {
+ laterRangeExists = true;
+ break outer;
+ }
+ }
+ }
+ if (laterRangeExists) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+ }
+ }
+
+ int productiveStart = prefixSlots;
+ int maxProductiveLen = maxProductiveEnd - productiveStart;
+ if (maxProductiveLen <= 0) {
+ return everything();
+ }
+
+ // Classify each dim in [productiveStart, maxProductiveEnd) as "pinned" or not.
+ // A dim is pinned iff every space has the same single-key value on that dim.
+ //
+ // Only trailing-pinned dims are split out into their own slots; leading-pinned
+ // dims are folded into the compound. V1's shape works this way: when all spaces
+ // agree on an equality on some leading dim(s), the compound bytes anchor the
+ // scan tightly with that prefix. But when pinned equalities appear after an
+ // unbounded-range slot (i.e., trailing-pinned), they don't narrow the scan's
+ // start/stop bounds any further — V1 emits them as separate slots past the
+ // unbounded-compound slot, and ScanRanges.getBoundPkColumnCount() correctly
+ // stops counting at the first unbounded slot.
+ //
+ // Example: (pk1, pk2) > ('0','0') AND pk3 = '...' AND pk4 = '...' →
+ // compound spans pk1+pk2 with unbounded upper; pk3 and pk4 become trailing
+ // pinned slots. Bound count stops at the compound (3 cols total with tenantId).
+ KeyRange[] pinnedValue = new KeyRange[maxProductiveEnd];
+ for (int d = productiveStart; d < maxProductiveEnd; d++) {
+ KeyRange shared = null;
+ boolean allAgree = true;
+ for (KeySpace ks : list.spaces()) {
+ KeyRange r = ks.get(d);
+ if (
+ !r.isSingleKey() || r == KeyRange.IS_NULL_RANGE || r == KeyRange.IS_NOT_NULL_RANGE
+ ) {
+ allAgree = false;
+ break;
+ }
+ if (shared == null) {
+ shared = r;
+ } else if (!shared.equals(r)) {
+ allAgree = false;
+ break;
+ }
+ }
+ pinnedValue[d] = allAgree ? shared : null;
+ }
+ // Compound window: [compoundStart, compoundEnd). Leading-pinned dims stay in
+ // the compound (compoundStart = productiveStart); trailing-pinned dims are
+ // split out only when the compound has at least one non-pinned range dim AND
+ // the compound would end up with an unbounded side. Splitting otherwise would
+ // break scans where the compound captures all narrowing in a single fully-
+ // bounded range (e.g., `id LIKE 'xy%' AND type = 1` → one 2-col compound
+ // with both bounds fully specified).
+ int compoundStart = productiveStart;
+ int compoundEnd = maxProductiveEnd;
+ // Check whether splitting is warranted: find the first non-pinned dim. If all
+ // dims are pinned or there's no non-pinned dim before the trailing pinned
+ // run, don't split.
+ int firstNonPinned = -1;
+ for (int d = productiveStart; d < maxProductiveEnd; d++) {
+ if (pinnedValue[d] == null) {
+ firstNonPinned = d;
+ break;
+ }
+ }
+ if (firstNonPinned >= 0) {
+ // Check whether the non-pinned dim(s) would produce a compound with an
+ // unbounded side across any space. Only then is trailing-split beneficial.
+ boolean anyUnbound = false;
+ for (KeySpace ks : list.spaces()) {
+ for (int d = firstNonPinned; d < maxProductiveEnd && pinnedValue[d] == null; d++) {
+ KeyRange r = ks.get(d);
+ if (r == KeyRange.EVERYTHING_RANGE || r.isUnbound()) {
+ anyUnbound = true;
+ break;
+ }
+ }
+ if (anyUnbound) break;
+ }
+ if (anyUnbound) {
+ while (compoundEnd > compoundStart && pinnedValue[compoundEnd - 1] != null) {
+ compoundEnd--;
+ }
+ }
+ }
+ int compoundLen = compoundEnd - compoundStart;
+
+ // Safety gate: compound emission is UNSAFE when any space has a non-single-key dim
+ // followed by ANY further constraint (pinned or range) on a later dim WITHIN THE
+ // COMPOUND WINDOW. The compound byte range [lo1+lo2, hi1+hi2) is lex-wider than the
+ // conjunction, so rows with col1 strictly between lo1 and hi1 pass regardless of
+ // col2's value — and V2 doesn't emit a residual SkipScanFilter to reject them. V1
+ // falls back to per-column projection with a SkipScanFilter for this shape.
+ //
+ // Example broken shapes:
+ // key_1 in [000,200) AND key_2 in [aabb,aadd) → rows with key_1='100', key_2='aaaa'
+ // are in the compound [000aabb, 200) but shouldn't match (key_2 out of range).
+ // CREATETIME in [A,B] AND ACCOUNTID='v' → rows with any ACCOUNTID value in the
+ // middle CREATETIME band are in the compound but shouldn't match.
+ //
+ // Checked within compound window: trailing pinned dims outside the window are split
+ // into separate slots and don't participate in this check.
+ //
+ // Rule: if any space has a non-single-key dim followed by any further constrained
+ // dim (single-key or range) in the compound window, fall back. Trailing non-single-
+ // key within the window is safe (last dim of the compound range, bound correctly).
+ for (KeySpace ks : list.spaces()) {
+ boolean sawNonSingleKey = false;
+ for (int d = compoundStart; d < compoundEnd; d++) {
+ KeyRange dim = ks.get(d);
+ if (dim == KeyRange.EVERYTHING_RANGE) continue;
+ if (!dim.isSingleKey()) {
+ if (sawNonSingleKey) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+ sawNonSingleKey = true;
+ } else if (sawNonSingleKey) {
+ return emitV1Projection(list, nPkColumns, cartesianBound, prefixSlots);
+ }
+ }
+ }
+
+ // Build one compound KeyRange per space, only over the [compoundStart, compoundEnd)
+ // window. Pinned prefix/suffix dims are emitted as individual slots outside the loop.
+ List<KeyRange> compounds = new ArrayList<>(list.size());
+ // Skip the compound build entirely when every productive dim is pinned: no range
+ // part to compound. The pinned slots below carry all the narrowing.
+ if (compoundLen > 0) {
+ for (KeySpace ks : list.spaces()) {
+ int end = firstProductiveStop(ks, prefixSlots);
+ // Clamp end to the compound window: trailing pinned dims are emitted separately.
+ if (end > compoundEnd) end = compoundEnd;
+ // Per-dim view: dims [compoundStart, end) as individual slots with slotSpan 0.
+ int len = end - compoundStart;
+ if (len <= 0) {
+ // Space is all-EVERYTHING past the prefix — contributes EVERYTHING. The whole
+ // list's emission becomes EVERYTHING.
+ return everything();
+ }
+ List<List<KeyRange>> perDimSlots = new ArrayList<>(len);
+ int[] perDimSpan = new int[len];
+ boolean allSingleKey = true;
+ // IS_NULL_RANGE has empty bounds and KeyRange.isSingleKey() returns true. For
+ // a non-leading IS NULL with trailing unconstrained PK columns AND leading
+ // single-key equality prefix, the compound must be half-open to exclude rows with
+ // non-null values on the null-dim. For leading IS NULL (no single-key prefix),
+ // keeping the IS_NULL_RANGE sentinel lets ScanRanges.create handle it specially
+ // (it has separate codepaths for IS_NULL_RANGE that set the right scan bounds).
+ boolean hasTrailingUnconstrained = end < ks.nDims();
+ // Count the leading single-key equality prefix within this space's productive run.
+ int leadingSingleKeyCount = 0;
+ for (int d = compoundStart; d < end; d++) {
+ KeyRange dim = ks.get(d);
+ if (dim.isSingleKey() && dim != KeyRange.IS_NULL_RANGE
+ && dim != KeyRange.IS_NOT_NULL_RANGE) {
+ leadingSingleKeyCount++;
+ } else {
+ break;
+ }
+ }
+ for (int d = compoundStart; d < end; d++) {
+ KeyRange dim = ks.get(d);
+ perDimSlots.add(Collections.singletonList(dim));
+ if (!dim.isSingleKey()) {
+ allSingleKey = false;
+ } else if ((dim == KeyRange.IS_NULL_RANGE || dim == KeyRange.IS_NOT_NULL_RANGE)
+ && hasTrailingUnconstrained && leadingSingleKeyCount > 0) {
+ // Non-leading IS NULL with leading equality prefix: convert to half-open so
+ // trailing non-null rows don't sneak in via the nextKey-bumped upper.
+ allSingleKey = false;
+ }
+ }
+ // Use setKey variant with schemaStartIndex so the schema is walked starting from
+ // the user-tail fields (after prefix columns like salt, viewIndexId, tenantId).
+ // Without this, the first user-tail slot's bytes get decoded against the schema's
+ // leading field (e.g. the VARCHAR tenantId slot), which appends a spurious `\x00`
+ // separator for non-fixed-width leading fields.
+ byte[] lo = getKeyWithSchemaOffset(schema, perDimSlots, perDimSpan,
+ KeyRange.Bound.LOWER, compoundStart);
+ byte[] hi = getKeyWithSchemaOffset(schema, perDimSlots, perDimSpan,
+ KeyRange.Bound.UPPER, compoundStart);
+ // Strip the trailing separator byte for the last productive field if it's
+ // variable-length AND that field is the last field in the full PK schema.
+ // ScanUtil.getMinKey/getMaxKey append a trailing separator for variable-length
+ // fields (both ASC `\x00` and DESC `\xFF`). Downstream ScanRanges.create ->
+ // ScanUtil.setKey walks our compound bytes again and re-appends another separator
+ // when it finishes the same field, producing a double-separator bug (extra
+ // trailing `\xFF` for DESC, extra `\x00` for ASC). Stripping here lets the
+ // downstream setKey re-add it correctly.
+ //
+ // IMPORTANT: only strip when the last productive field is actually the last field
+ // in the PK. If there are unconstrained PK fields after the productive run, the
+ // trailing separator is an internal boundary marker between the last-productive
+ // dim and the (wildcard) next dim — downstream setKey needs it to know where the
+ // constrained prefix ends. Stripping in that case produces a startRow that's too
+ // short and misses the dim boundary (see QueryCompilerTest.testRVCScanBoundaries1).
+ org.apache.phoenix.schema.ValueSchema.Field lastField =
+ schema.getField(compoundStart + len - 1);
+ boolean lastIsVarLength = !lastField.getDataType().isFixedWidth();
+ boolean lastIsLastPkField = (compoundStart + len) == schema.getMaxFields();
+ // Strip when:
+ // (a) this field is the last PK field (no trailing unconstrained dims), OR
+ // (b) all productive dims are single-key (we'll emit as a point key, and the
+ // trailing separator is redundant — downstream SkipScanFilter works with
+ // raw point bytes).
+ // When neither condition holds (range with trailing EVERYTHING dims), keep the
+ // separator as a boundary marker for downstream setKey (see testRVCScanBoundaries1).
+ if (lastIsVarLength && (lastIsLastPkField || allSingleKey)) {
+ lo = stripTrailingSeparator(lo, lastField);
+ hi = stripTrailingSeparator(hi, lastField);
+ }
+ // Wrap into a compound KeyRange. getMinKey/getMaxKey already apply exclusive-bound
+ // bumping internally.
+ //
+ // For all-single-key compounds (every productive dim is a point equality), emit as
+ // KeyRange.getKeyRange(bytes) — a single-key range. Downstream
+ // ScanRanges.isPointLookup() needs isSingleKey()=true on the range to promote the
+ // scan to a proper GET-style point lookup; a half-open [lo, hi) range never
+ // qualifies even when lo and hi are nextKey-adjacent.
+ //
+ // EXCEPTION: when this space's productive dims end before maxProductiveEnd (i.e. the
+ // slot-span covers more dims than this space constrains), a single-key compound would
+ // have fewer bytes than the SkipScanFilter expects for this slot. Emit a half-open
+ // range [lo, nextKey(lo)) in that case so the range matches any row whose leading
+ // bytes equal lo — the trailing unconstrained dims are implicitly wild.
+ KeyRange compound;
+ boolean shorterThanSlotSpan = end < compoundEnd;
+ if (allSingleKey && lo != null && lo.length > 0 && !shorterThanSlotSpan) {
+ compound = KeyRange.getKeyRange(lo);
+ } else {
+ compound = KeyRange.getKeyRange(lo == null ? KeyRange.UNBOUND : lo, true,
+ hi == null ? KeyRange.UNBOUND : hi, false);
+ }
+ if (compound == KeyRange.EMPTY_RANGE) {
+ continue;
+ }
+ compounds.add(compound);
+ }
+ if (compounds.isEmpty()) {
+ return nothing();
+ }
+
+ // Cartesian bound: if the number of compound ranges exceeds the bound, we need to
+ // widen. That widening happens upstream in KeySpaceList; by the time we reach here
+ // the list is already bounded. Still, apply a defensive cap.
+ BigInteger bound = BigInteger.valueOf(Math.max(1, cartesianBound));
+ if (BigInteger.valueOf(compounds.size()).compareTo(bound) > 0) {
+ // Over budget — drop everything past the bound (sound widening: truncation admits
+ // more rows but never fewer; residual filter handles any extras).
+ compounds = compounds.subList(0, cartesianBound);
Review Comment:
Truncating this list narrows the union; it does not widen it. Any matching compound range after the cutoff disappears from the scan start/stop envelope and SkipScanFilter, producing false negatives that a residual filter cannot recover. When over budget, replace discarded ranges with a covering superset (or fall back to an unbounded/per-slot scan) rather than taking the first entries.
This issue also appears on line 980 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@phoenix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org