You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "xuzifu666 (via GitHub)" <gi...@apache.org> on 2026/09/26 11:12:54 UTC
[PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
xuzifu666 opened a new pull request, #5295:
URL: https://github.com/apache/calcite/pull/5295
jira: https://issues.apache.org/jira/browse/CALCITE-5907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4124162417
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
> I hope you have checked that this works for any nullability
Good point @mihaibudiu . The check only looks at the type name, not nullability, and it is safe for both cases: =(x, TRUE) → x with nullable x (if x is NULL, =(NULL, TRUE) and bare NULL are both UNKNOWN, i.e. FALSE under UNKNOWN_AS_FALSE) and with NOT NULL x (trivially equivalent).
I had added NOT NULL variants of VARCHAR, INTEGER and BOOLEAN to testSimplifyAndEqualityTrue to cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-5891496124
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-5891591959
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-6062546352
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4124950334
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Or call a 'isWellFormed' method before entering the loop. It sends a strong message that "this code doesn't run on garbage".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-6014163464
> The same defect remains when the operands are swapped. `simplifyComparison` checks only that `o0` is BOOLEAN ([RexSimplify.java#L751](https://github.com/apache/calcite/blob/f83d5ef65a442b62b26c74cfe7d6c0c1c85922d6/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L751)), and `Comparison.of` then swaps the operands so that `cmp.ref` is the VARCHAR one. With this PR applied, in every `RexUnknownAs` mode:
>
> Input Result
> `=(true, ?0.varchar0)` `?0.varchar0`
> `AND(=(true, ?0.varchar0), =(true, ?0.varchar1))` `AND(?0.varchar0, ?0.varchar1)`, of type VARCHAR
> `AND(=(false, ?0.varchar0), =(false, ?0.varchar1))` `AND(NOT(?0.varchar0), NOT(?0.varchar1))`
> `AND(<>(false, ?0.varchar0), <>(false, ?0.varchar1))` `AND(?0.varchar0, ?0.varchar1)`
> A front end that builds `=(TRUE, c)` without a cast, as Flink does in the JIRA, still gets the plan [CALCITE-5907](https://issues.apache.org/jira/browse/CALCITE-5907) reports for `WHERE TRUE = c AND TRUE = d`. `testSimplifyAndEqualityTrue` needs these four inputs too.
>
> The JIRA summary, and with it the PR title and the squashed commit, does not name the symptom, and the bug is wider than `AND`. Something like `RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand` covers the swapped and FALSE forms as well.
Good catch, thank you @vlsi ! The swapped case indeed slipped through: simplifyComparison only checked o0 and Comparison.of happily swapped the operands, so =(TRUE, c) still collapsed to c. Fixed per your suggestion — there is now a private isWellFormed(left, right) helper (same type name on both sides), called both from the AND loop and from simplifyComparison before the BOOLEAN branch, so it now guards both directions and the = false / <> false forms. testSimplifyAndEqualityTrue covers the four inputs from your table (as checkSimplifyUnchanged), plus AND(=($0, true), ?0.bool1) with $0 of type ANY; the non-BOOLEAN inputs run with withParanoid(false) so a regression reports the simplified expression rather than a ClassCastException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4112335242
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
This is bullshit.
CALCITE-5907 is invalid, because you should never have "varchar-column = bool-literal" in a Rex node. If we see non-boolean terms in a boolean expression, we should throw.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4124102342
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Thanks for the suggestion @julianhyde . `isAlwaysTrue` / `isAlwaysFalse` are already defensive: both `RexLiteral` and `RexCall` return false unless the expression has BOOLEAN type, so the string literal 'true' already returns false.The call-site check guards the other operand (the one that would replace the =` term), so it can't be pushed intoisAlwaysTrue. Without it,AND(=(x, TRUE), ...)with VARCHARxwould simplify toAND(x, ...) which is the CALCITE-5907 bug. Please correct me if I am wrong~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "vlsi (via GitHub)" <gi...@apache.org>.
vlsi commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-6069057630
@xuzifu666 I squashed the branch into one commit, 2ff3fb99c2 and updated the PR description. The code is the same as in 939e7c1ce7.
I guess we can merge this once CI succeeds.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-6060378578
@vlsi Are you happy with current changes? if there no other objections I would merge it in few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4124102342
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Thanks for the suggestion @julianhyde . `isAlwaysTrue` / `isAlwaysFalse` are already defensive: both `RexLiteral` and `RexCall` return false unless the expression has BOOLEAN type, so the string literal 'true' already returns false.The call-site check guards the other operand (the one that would replace the =` term), so it can't be pushed intoisAlwaysTrue. Without it,AND(=(x, TRUE), ...)with VARCHAR x would simplify toAND(x, ...) which is the CALCITE-5907 bug. Please correct me if I am wrong~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-5874188677
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [66.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4113880154
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Let me share my understanding of this part—though I might be mistaken: I agree that a well-typed Rex tree should never contain =(varchar, bool), and that SqlValidator is the right place to enforce comparability. However, that invariant is not enforced at the Rex layer itself: `RexBuilder.makeCall(EQUALS, ...)` performs no operand-family check (`EQUALS`' return-type inference is unconditionally `BOOLEAN`), so downstream systems that construct expressions programmatically (e.g. Flink, the reporter of https://issues.apache.org/jira/browse/FLINK-27402) can and do produce such nodes.
The bug here is not the input's existence, but that the simplifier turned a valid `BOOLEAN` condition into a non-`BOOLEAN` one. A rewrite rule must never produce output less valid than its input; leaving the equality untouched is the minimal, safe behavior. Throwing from a simplification hot path would turn a silent robustness gap into planner crashes for rels that currently plan and execute fine, and conflates rewriting with validation, Calcite keeps those separate (cf. `RexChecker`/`RexUtil.verify`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4220229203
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,61 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
+ @Test void testSimplifyAndEqualityTrue() {
+ // "=(x, true)" simplifies to "x" only if x is BOOLEAN;
+ // otherwise the AND would get a non-BOOLEAN operand
+ //
+ // Paranoid mode throws from RexSimplify.verify for these non-BOOLEAN
+ // inputs, so turn it off to let checkSimplifyUnchanged report the
+ // simplified expression instead
+ simplify = this.simplify.withParanoid(false);
+ checkSimplifyUnchanged(
+ and(eq(vVarchar(0), trueLiteral),
+ eq(vVarchar(1), trueLiteral)));
+ checkSimplifyUnchanged(
+ and(eq(vInt(0), trueLiteral),
+ eq(vInt(1), trueLiteral)));
+ // same, with NOT NULL variables
+ checkSimplifyUnchanged(
+ and(eq(vVarcharNotNull(0), trueLiteral),
+ eq(vVarcharNotNull(1), trueLiteral)));
+ checkSimplifyUnchanged(
+ and(eq(vIntNotNull(0), trueLiteral),
+ eq(vIntNotNull(1), trueLiteral)));
+ // literal on the left (swapped) form
+ checkSimplifyUnchanged(eq(trueLiteral, vVarchar(0)));
+ checkSimplifyUnchanged(
+ and(eq(trueLiteral, vVarchar(0)),
+ eq(trueLiteral, vVarchar(1))));
+ checkSimplifyUnchanged(
+ and(eq(falseLiteral, vVarchar(0)),
+ eq(falseLiteral, vVarchar(1))));
+ checkSimplifyUnchanged(
+ and(ne(falseLiteral, vVarchar(0)),
+ ne(falseLiteral, vVarchar(1))));
+ // "=$0, true" where $0 has type ANY is no longer simplified to "$0";
+ // ITEM on a MAP<VARCHAR, ANY> column (e.g. MongoDB's _MAP) produces ANY,
+ // which may be BOOLEAN at runtime
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4111935021
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
I hope you have checked that this works for any nullability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "vlsi (via GitHub)" <gi...@apache.org>.
vlsi commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4189685779
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,6 +2013,10 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
+ if (call.getOperands().get(0).getType().getSqlTypeName()
+ != call.getOperands().get(1).getType().getSqlTypeName()) {
+ break;
+ }
Review Comment:
Julian suggested an `isWellFormed` check ([comment](https://github.com/apache/calcite/pull/5295#discussion_r4124950334)). A helper of that shape, called here and from `simplifyComparison` before L751, would also stop the swapped-operand case from the review body, and its name would say why the term is left alone. As written, the guard exists only in this loop and carries no comment.
This also changes plans for `ANY`: under `UNKNOWN_AS_FALSE`, `AND(=($0, true), ?0.bool1)` with `$0` of type `ANY` used to become `AND($0, ?0.bool1)` and now stays unchanged. It needs a mention in the JIRA and a test, since `ITEM` on a `MAP<VARCHAR, ANY>` column, such as MongoDB's `_MAP`, produces `ANY`.
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,35 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
+ @Test void testSimplifyAndEqualityTrue() {
+ // "=(x, true)" can be simplified to "x" only if x has BOOLEAN type;
+ // there is no implicit cast from VARCHAR or INTEGER, so the equality
+ // must be retained
Review Comment:
The stated reason is not accurate: Calcite does coerce VARCHAR and numeric types to BOOLEAN (`SqlTypeCoercionRule`). The equality has to stay because replacing it with `x` would put a non-BOOLEAN operand into the AND. Suggest: `"=(x, true)" simplifies to "x" only if x is BOOLEAN; otherwise the AND would get a non-BOOLEAN operand.`
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,35 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
+ @Test void testSimplifyAndEqualityTrue() {
+ // "=(x, true)" can be simplified to "x" only if x has BOOLEAN type;
+ // there is no implicit cast from VARCHAR or INTEGER, so the equality
+ // must be retained
+ checkSimplifyUnchanged(
+ and(eq(vVarchar(0), trueLiteral),
+ eq(vVarchar(1), trueLiteral)));
Review Comment:
With paranoid mode on (`RexProgramBuilderBase` turns it on for every test), a regression here throws `ClassCastException` from `RexInterpreter` inside `RexSimplify.verify` and never shows the simplified expression. Turning paranoid mode off for the non-BOOLEAN inputs, as `testSimplifyItemRangeTerms` does with `simplify = simplify.withParanoid(false)`, makes `checkSimplifyUnchanged` print the expression it got instead.
Nit: the second argument of `and(` is indented at the same level as `and(` itself, so it reads like a second argument of `checkSimplifyUnchanged`; the same applies at L2487, L2494, L2497, and L2500.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4133941858
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Good idea, done @julianhyde . The loop now starts with a well-formedness guard: unless both operands of = have the same type name (e.g.=(x, TRUE) with `VARCHAR` or `INTEGER` x is malformed), it breaks without simplifying. This is equivalent to the previous check, sinceisAlwaysTrue() can only be true for a BOOLEAN expression, but it states the precondition up front. I compare `getSqlTypeName()` rather than `RelDataType.equals`, because `TRUE` is `NOT NULL` and equals would compare nullability, blocking the valid simplification of=(nullable Bool X, TRUE).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4124934147
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Maybe just check that the operands have the same type? Your change is making RexSimplify more complicated, because we are accommodating a client that is sending us garbage. That is tech debt right there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4220222658
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -703,6 +703,18 @@ private static boolean containsEquivalentComparison(List<RexNode> nodes,
return false;
}
+ /** Returns whether a comparison between {@code left} and {@code right}
+ * is well-formed, that is, its operands have the same type.
+ *
+ * <p>Comparisons whose operands have different types (e.g. a VARCHAR
+ * column compared with the BOOLEAN literal TRUE) never result from
+ * Calcite's own SQL translation, but clients that build Rex trees
+ * programmatically sometimes create them; such comparisons are left
+ * unsimplified. */
+ private static boolean isWellFormed(RexNode left, RexNode right) {
Review Comment:
Thanks for your suggestion: I renamed the helper to isBooleanComparison(left, right) with your javadoc (the unverified provenance claim is gone; at simplifyComparison the call collapsed to a single condition, behavior unchanged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4220225303
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,61 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
Review Comment:
Done.
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,61 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
+ @Test void testSimplifyAndEqualityTrue() {
+ // "=(x, true)" simplifies to "x" only if x is BOOLEAN;
+ // otherwise the AND would get a non-BOOLEAN operand
+ //
+ // Paranoid mode throws from RexSimplify.verify for these non-BOOLEAN
+ // inputs, so turn it off to let checkSimplifyUnchanged report the
+ // simplified expression instead
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-6031702284
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=5295&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=5295&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=5295&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=5295)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "julianhyde (via GitHub)" <gi...@apache.org>.
julianhyde commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4116412685
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Ah, it's a malformed `=`, not a malformed boolean term in `AND` or `OR`. That's not OK, but it's understandable. I think it would break a lot of clients it Calcite started to reject such `=` calls.
So I think you should make `isAlwaysTrue` more defensive - push the 'type is boolean' check into that method. Thus for the string literal 'true', `isAlwaysTrue` will return false.
Ditto `isAlwaysFalse` and any analogous methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4133941858
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -2013,11 +2013,13 @@ private <C extends Comparable<C>> RexNode simplifyAnd2ForUnknownAsFalse(
// Simplify BOOLEAN expressions if possible
while (term.getKind() == SqlKind.EQUALS) {
RexCall call = (RexCall) term;
- if (call.getOperands().get(0).isAlwaysTrue()) {
+ if (call.getOperands().get(0).isAlwaysTrue()
+ && call.getOperands().get(1).getType().getSqlTypeName() == SqlTypeName.BOOLEAN) {
Review Comment:
Good idea, done @julianhyde . The loop now starts with a well-formedness guard: unless both operands of =` have the same type name (e.g.=(x, TRUE) with VARCHAR or INTEGER x is malformed), it breaks without simplifying. This is equivalent to the previous check, sinceisAlwaysTrue() can only be true for a BOOLEAN expression, but it states the precondition up front. I compare `getSqlTypeName()` rather than `RelDataType.equals`, because `TRUE` is `NOT NULL` and equals would compare nullability, blocking the valid simplification of=(nullable Bool X, TRUE).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] Unexpected boolean expression simplification for And expression [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on PR #5295:
URL: https://github.com/apache/calcite/pull/5295#issuecomment-5903001238
Since this pr is approved and should had addressed comments, if no other objections I would merge it in few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 merged PR #5295:
URL: https://github.com/apache/calcite/pull/5295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-5907] RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand [calcite]
Posted by "vlsi (via GitHub)" <gi...@apache.org>.
vlsi commented on code in PR #5295:
URL: https://github.com/apache/calcite/pull/5295#discussion_r4219720981
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,61 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
+ @Test void testSimplifyAndEqualityTrue() {
+ // "=(x, true)" simplifies to "x" only if x is BOOLEAN;
+ // otherwise the AND would get a non-BOOLEAN operand
+ //
+ // Paranoid mode throws from RexSimplify.verify for these non-BOOLEAN
+ // inputs, so turn it off to let checkSimplifyUnchanged report the
+ // simplified expression instead
Review Comment:
Paranoid mode throws only when the simplified expression evaluates differently from the original, not for these inputs as such. Suggest: `// With paranoid mode on, a wrong simplification of these non-BOOLEAN inputs fails inside RexSimplify.verify with an exception that hides the result; turn it off so that checkSimplifyUnchanged prints the simplified expression`.
##########
core/src/main/java/org/apache/calcite/rex/RexSimplify.java:
##########
@@ -703,6 +703,18 @@ private static boolean containsEquivalentComparison(List<RexNode> nodes,
return false;
}
+ /** Returns whether a comparison between {@code left} and {@code right}
+ * is well-formed, that is, its operands have the same type.
+ *
+ * <p>Comparisons whose operands have different types (e.g. a VARCHAR
+ * column compared with the BOOLEAN literal TRUE) never result from
+ * Calcite's own SQL translation, but clients that build Rex trees
+ * programmatically sometimes create them; such comparisons are left
+ * unsimplified. */
+ private static boolean isWellFormed(RexNode left, RexNode right) {
Review Comment:
The javadoc says that comparisons whose operands have different types "never result from Calcite's own SQL translation". Nothing in `RexSimplify` or its tests backs this, and no change to the converter would update it. Both call sites need only "both operands are BOOLEAN": in `simplifyComparison` `o0` is already BOOLEAN, and in the `AND` loop only a BOOLEAN node can be `isAlwaysTrue()`. A helper named for that check, such as `isBooleanComparison(left, right)` with the javadoc `Returns whether both operands of a comparison have type BOOLEAN.`, behaves the same at both sites and needs no claim about where such trees come from. `isWellFormed` names a general property, so a later caller could use it to skip valid simplifications such as INTEGER compared with BIGINT. The first sentence also says "the same type" where the check compares `SqlTypeName`.
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,61 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
+ @Test void testSimplifyAndEqualityTrue() {
+ // "=(x, true)" simplifies to "x" only if x is BOOLEAN;
+ // otherwise the AND would get a non-BOOLEAN operand
+ //
+ // Paranoid mode throws from RexSimplify.verify for these non-BOOLEAN
+ // inputs, so turn it off to let checkSimplifyUnchanged report the
+ // simplified expression instead
+ simplify = this.simplify.withParanoid(false);
+ checkSimplifyUnchanged(
+ and(eq(vVarchar(0), trueLiteral),
+ eq(vVarchar(1), trueLiteral)));
+ checkSimplifyUnchanged(
+ and(eq(vInt(0), trueLiteral),
+ eq(vInt(1), trueLiteral)));
+ // same, with NOT NULL variables
+ checkSimplifyUnchanged(
+ and(eq(vVarcharNotNull(0), trueLiteral),
+ eq(vVarcharNotNull(1), trueLiteral)));
+ checkSimplifyUnchanged(
+ and(eq(vIntNotNull(0), trueLiteral),
+ eq(vIntNotNull(1), trueLiteral)));
+ // literal on the left (swapped) form
+ checkSimplifyUnchanged(eq(trueLiteral, vVarchar(0)));
+ checkSimplifyUnchanged(
+ and(eq(trueLiteral, vVarchar(0)),
+ eq(trueLiteral, vVarchar(1))));
+ checkSimplifyUnchanged(
+ and(eq(falseLiteral, vVarchar(0)),
+ eq(falseLiteral, vVarchar(1))));
+ checkSimplifyUnchanged(
+ and(ne(falseLiteral, vVarchar(0)),
+ ne(falseLiteral, vVarchar(1))));
+ // "=$0, true" where $0 has type ANY is no longer simplified to "$0";
+ // ITEM on a MAP<VARCHAR, ANY> column (e.g. MongoDB's _MAP) produces ANY,
+ // which may be BOOLEAN at runtime
Review Comment:
`"=$0, true"` is missing its parentheses. "is no longer simplified" describes the change rather than the rule, and "which may be BOOLEAN at runtime" gives the opposite reason: the term stays because a value of type `ANY` need not be BOOLEAN. Suggest: `// "=($0, true)" stays when $0 has type ANY, since its value need not be BOOLEAN; ITEM on a MAP<VARCHAR, ANY> column, such as MongoDB's _MAP, has type ANY`.
##########
core/src/test/java/org/apache/calcite/rex/RexProgramTest.java:
##########
@@ -2472,6 +2472,61 @@ private void checkExponentialCnf(int n) {
"=(?0.notNullInt0, ?0.int1)");
}
+ /** Test case for
+ * <a href="https://issues.apache.org/jira/browse/CALCITE-5907">[CALCITE-5907]
+ * Unexpected boolean expression simplification for And expression</a>. */
Review Comment:
The link text should match the JIRA summary, which is now `RexSimplify replaces a comparison with a BOOLEAN literal by a non-BOOLEAN operand`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org