You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/27 23:52:56 UTC

[GitHub] [spark] sunchao opened a new pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

sunchao opened a new pull request #29565:
URL: https://github.com/apache/spark/pull/29565


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   Currently, in cases like the following:
   ```sql
   SELECT * FROM t WHERE age < 40
   ```
   where `age` is of short type, Spark won't be able to simplify this and can only generate filter `cast(age, int) < 40`. This won't get pushed down to datasources and therefore is not optimized.
   
   This PR proposes a optimizer rule to improve this when the following constraints are satisfied:
    - input expression is binary comparisons when one side is a cast operation and another is a literal. 
    - both the cast child expression and literal are of integral type (i.e., byte, short, int or long)
   
   When this is true, it tries to do several optimizations to either simplify the expression or move the cast to the literal side, so
   result filter for the above case becomes `age < cast(40 as smallint)`. This is better since the cast can be optimized away later and the filter can be pushed down to data sources.
   
   The approach this PR uses references a similar effort in Presto (https://prestosql.io/blog/2019/05/21/optimizing-the-casts-away.html). Here we only handles integral types but plan to extend to other types as follow-ups.
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   As mentioned in the previous section, when cast is not optimized, it cannot be pushed down to data sources which can lead
   to unnecessary IO and therefore longer job time and waste of resources. This helps to improve that.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   No. 
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   
   Added unit tests for both the optimizer rule and filter pushdown on datasource level for both Orc and Parquet.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682285619


   cc @dbtsai @holdenk @dongjoon-hyun @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704215



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {

Review comment:
       @sunchao . BTW, why do we test `DataSourceV2` only?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385046






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683383762


   **[Test build #128033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128033/testReport)** for PR 29565 at commit [`1a7a3e1`](https://github.com/apache/spark/commit/1a7a3e1f1ae6d87b390035423ebeabca0a163ae9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691210680


   **[Test build #128577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128577/testReport)** for PR 29565 at commit [`de911da`](https://github.com/apache/spark/commit/de911da549fe7c39aed2b2ae92a100d9c1fff2ed).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480473760



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType))

Review comment:
       Maybe just do 
   ```
   BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
   ```
   to make sure, and it will be more readable as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486063309



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(

Review comment:
       nit: `simplifyIntegralComparison`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385518


   **[Test build #128035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128035/testReport)** for PR 29565 at commit [`1a7a3e1`](https://github.com/apache/spark/commit/1a7a3e1f1ae6d87b390035423ebeabca0a163ae9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685119528


   **[Test build #128165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128165/testReport)** for PR 29565 at commit [`aec3053`](https://github.com/apache/spark/commit/aec3053103a0d9448769722131abd046ec4ac20b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480476764



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:

Review comment:
       ```
   if `exp > min of exp type > value`, the cases are of following:
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479700098



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       Shall we rename to more precise one like `UnwrapIntegralCast` because this optimizer only handle `IntegralType`? We can use `UnwrapCast` later when we support more types like `FractionalType`, `Date`, `String` and so on.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480850745



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>

Review comment:
       I do plan to add more checks in this method, for instance checking whether a bigint can be cast to float/double based on the digits. I'm fine to remove it for now and add it back later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691807275


   Late LGTM with one minor comment: https://github.com/apache/spark/pull/29565/files#r487646973


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685171506


   **[Test build #128157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128157/testReport)** for PR 29565 at commit [`b5371c5`](https://github.com/apache/spark/commit/b5371c59cf44bf1c0e70571dda7736a4f72bcd23).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486663619



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparisonSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans.DslLogicalPlan
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.IntegralLiteralTestUtils._
+import org.apache.spark.sql.catalyst.expressions.aggregate.First
+import org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types.{BooleanType, DoubleType, IntegerType}
+
+class UnwrapCastInBinaryComparisonSuite extends PlanTest {
+
+  object Optimize extends RuleExecutor[LogicalPlan] {
+    val batches: List[Batch] =
+      Batch("Unwrap casts in binary comparison", FixedPoint(10),
+        NullPropagation, ConstantFolding, UnwrapCastInBinaryComparison) :: Nil
+  }
+
+  val testRelation: LocalRelation = LocalRelation('a.short, 'b.float)
+
+  test("unwrap casts when literal == max") {
+    val v = Short.MaxValue
+    assertEquivalent('a > v.toInt, 'a.attr.falseIfNotNull)
+    assertEquivalent('a >= v.toInt, 'a === v)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, 'a.attr.trueIfNotNull)
+    assertEquivalent('a < v.toInt, 'a =!= v)
+  }
+
+  test("unwrap casts when literal > max") {
+    val v: Int = positiveInt
+    assertEquivalent('a > v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a >= v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a === v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a <=> v, false)
+    assertEquivalent('a <= v, 'a.attr.trueIfNotNull)
+    assertEquivalent('a < v, 'a.attr.trueIfNotNull)
+  }
+
+  test("unwrap casts when literal == min") {
+    val v = Short.MinValue
+    assertEquivalent('a > v.toInt, 'a =!= v)
+    assertEquivalent('a >= v.toInt, 'a.attr.trueIfNotNull)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, 'a === v)
+    assertEquivalent('a < v.toInt, 'a.attr.falseIfNotNull)
+  }
+
+  test("unwrap casts when literal < min") {
+    val v: Int = negativeInt
+    assertEquivalent('a > v, 'a.attr.trueIfNotNull)
+    assertEquivalent('a >= v, 'a.attr.trueIfNotNull)
+    assertEquivalent('a === v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a <=> v, false)
+    assertEquivalent('a <= v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a < v, 'a.attr.falseIfNotNull)
+  }
+
+  test("unwrap casts when literal is within range (min, max)") {
+    assertEquivalent('a > 300, 'a > 300.toShort)
+    assertEquivalent('a >= 500, 'a >= 500.toShort)
+    assertEquivalent('a === 32766, 'a === 32766.toShort)
+    assertEquivalent('a <=> 32766, 'a <=> 32766.toShort)
+    assertEquivalent('a <= -6000, 'a <= -6000.toShort)
+    assertEquivalent('a < -32767, 'a < -32767.toShort)
+  }
+
+  test("unwrap casts when cast is on rhs") {
+    val v = Short.MaxValue
+    assertEquivalent(Literal(v.toInt) < 'a, 'a.attr.falseIfNotNull)
+    assertEquivalent(Literal(v.toInt) <= 'a, Literal(v) === 'a)
+    assertEquivalent(Literal(v.toInt) === 'a, Literal(v) === 'a)
+    assertEquivalent(Literal(v.toInt) <=> 'a, Literal(v) === 'a)
+    assertEquivalent(Literal(v.toInt) >= 'a, 'a.attr.trueIfNotNull)
+    assertEquivalent(Literal(v.toInt) > 'a, 'a =!= v)
+
+    assertEquivalent(Literal(30) <= 'a, Literal(30.toShort) <= 'a)
+  }
+
+  test("unwrap cast should have no effect when input is not integral type") {
+    assertEquivalent('b > 42.0, Cast('b, DoubleType) > 42.0)
+    assertEquivalent('b >= 42.0, Cast('b, DoubleType) >= 42.0)
+    assertEquivalent('b === 42.0, Cast('b, DoubleType) === 42.0)
+    assertEquivalent('b <=> 42.0, Cast('b, DoubleType) <=> 42.0)
+    assertEquivalent('b <= 42.0, Cast('b, DoubleType) <= 42.0)
+    assertEquivalent('b < 42.0, Cast('b, DoubleType) < 42.0)
+    assertEquivalent(Literal(42.0) > 'b, Literal(42.0) > Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) >= 'b, Literal(42.0) >= Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) === 'b, Literal(42.0) === Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) <=> 'b, Literal(42.0) <=> Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) <= 'b, Literal(42.0) <= Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) < 'b, Literal(42.0) < Cast('b, DoubleType))
+  }
+
+  test("unwrap cast should skip when expression is non-deterministic") {
+    assertEquivalent(Cast(First('a, ignoreNulls = true), IntegerType) <=> positiveInt,
+      Cast(First('a, ignoreNulls = true), IntegerType) <=> positiveInt)
+    assertEquivalent(Cast(First('a, ignoreNulls = true), IntegerType) <=> negativeInt,
+      Cast(First('a, ignoreNulls = true), IntegerType) <=> negativeInt)

Review comment:
       👍 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp

Review comment:
       👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690892299






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485353841



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {

Review comment:
       We typically don't mix `if else` and patten matching. Maybe we can do nested patten matching or
   ```scala
   (minCmp.signum, maxCmp.signum, exp) match {
     case (_, 1,  EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) => ...
     ...
     case (_, 1, EqualNullSafe(_, _) if exp.deterministic => FalseLiteral
     case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) => fromExp.trueIfNotNull
     ...
     ...
   }
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479558357



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)

Review comment:
       nit, you might consider to add an implicit conversion rule in this class, so you can directly call `fromExp.falseIfNotNull`. Might be a little bit readable. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485354324



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.falseIfNotNull
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral

Review comment:
       @cloud-fan for safety, we will skip optimization if exp is non deterministic. WDYT?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487047097



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691333924


   **[Test build #128577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128577/testReport)** for PR 29565 at commit [`de911da`](https://github.com/apache/spark/commit/de911da549fe7c39aed2b2ae92a100d9c1fff2ed).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685119528


   **[Test build #128165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128165/testReport)** for PR 29565 at commit [`aec3053`](https://github.com/apache/spark/commit/aec3053103a0d9448769722131abd046ec4ac20b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478842634



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {

Review comment:
       This method is too long. Can we separate it to few smaller methods for better readability?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler

Review comment:
       nit: optimizes

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }

Review comment:
       If this rule targets predicate pushdown, shall we only capture the pattern `PhysicalOperation`? 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`

Review comment:
       Before I read the code below, I don't know what this doc means. Can you revise the example of "replaces the cast with different simpler constructs" a little?
   
   For example, `max` is ambiguous, actually it means the maximum value of `fromType`.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {

Review comment:
       For each of case, can we add few comment explaining what we're going to do.
   
   E.g.,
   
   ```scala
   // Because `value` is larger than maximum value of `fromType`, we can simplify binary comparison, e.g.,  non-null `fromExp` can never equal to `value`.
   if (maxCmp > 0) {
     ...
   }
   ```
   
   

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp // impossible but safe guard, same below
+        }
+      } else if (maxCmp == 0) {
+        exp match {
+          case GreaterThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(max, fromType)))
+          case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(max, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Compare lower bounds
+      val minCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, minInToType)
+      if (minCmp < 0) {
+        exp match {
+          case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+            return falseIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp
+        }
+      } else if (minCmp == 0) {
+        exp match {
+          case LessThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case GreaterThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(min, fromType)))
+          case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(min, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Now we can assume `value` is within the bound of the source type, e.g., min < value < max
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+
+    case _ => exp
+  }
+
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))
+
+  private[sql] def trueIfNotNull(e: Expression): Expression =
+    Or(IsNotNull(e), Literal(null, BooleanType))

Review comment:
       ditto.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`

Review comment:
       Is this actually `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`? That's said `Cast` and `Literal` must be same type.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`

Review comment:
       `Cast(fromExp, fromType)` -> `Cast(fromExp, toType)`?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp // impossible but safe guard, same below
+        }
+      } else if (maxCmp == 0) {
+        exp match {
+          case GreaterThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(max, fromType)))
+          case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(max, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Compare lower bounds
+      val minCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, minInToType)
+      if (minCmp < 0) {
+        exp match {
+          case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+            return falseIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp
+        }
+      } else if (minCmp == 0) {
+        exp match {
+          case LessThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case GreaterThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(min, fromType)))
+          case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(min, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Now we can assume `value` is within the bound of the source type, e.g., min < value < max
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+
+    case _ => exp
+  }
+
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))

Review comment:
       It's nicer to add a short comment, like "// Returns false if `e` is not null, otherwise returns null."




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485873707



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below

Review comment:
       Ur, the comment looks wrong. This case happens when `case (_, 1, EqualNullSafe(_, _)) if !exp.deterministic`, doesn't it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486513640



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))

Review comment:
       Yes agreed. I'll follow suggestion by @dongjoon-hyun and add handling of more types later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485864932



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))

Review comment:
       `IntegralType` check seems to be duplicated here and `canImplicitlyCast`. The current case match can be faster, but we had better keep single responsibility principle. In this case, I'd recommend you to handle it in `canImplicitlyCast` only. It will be more easier for maintenance and extending this rule for other types.
   ```scala
   -    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
   +    case BinaryComparison(Literal(_, toType), Cast(fromExp, _, _))
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486520084



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))

Review comment:
       @dongjoon-hyun @dbtsai I think we also need to check that:
   
   ```scala
   Cast(fromExp, dt1: _, _), Literal(value, dt2: _))
   ```
   `dt1` and `dt2` are actually the same type. I think this should be true always but we'd better to not rely on that assumption. Perhaps we need to pass both to `canImplicitlyCast` and check there.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479410585



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }

Review comment:
       Since we will add more rules, we will add more conditions for early return. Should we explicitly match the condition that can be simplified? For example,
   
   ```
       case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) if
         fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] && 
         Cast.canUpCast(fromType, toType)) =>
   ```
   Then we start the following code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479471849



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>

Review comment:
       Do you mean combining the cases like:
   ```scala
   exp match {
   case <case1> if maxCmp > 0 =>
   case <case2> if maxCmp == 0 =>
   case <case3> if minCmp < 0 =>
   case <case4> if minCmp == 0 =>
   ...
   ```
   ?. This will end up having many cases since they do not fully overlap with each other.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479700098



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       Shall we rename `UnwrapCast ` to more precise one like `UnwrapIntegralTypeCast` because this optimizer only handle `IntegralType`? We can use `UnwrapCast` later when we support more types like `FractionalType`, `Date`, `String` and so on. Also, you may want to mention `BinaryComparison` like `SimplifyBinaryComparison` because this optimizer only handles `BinaryComparison`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487648574



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.falseIfNotNull
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral

Review comment:
       SGTM




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691208083






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485345294



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       I was thinking for `value == max`, we can have a separate rule such as
   
   ```
    *  - `fromExp > value` where fromExp.type == value.dataType && getRange(fromExp)._2 == value` ==> if(isnull(fromExp), null, false)
   ...
   ...
   ```
   which should be valid for other types. Thus, we don't need to handle `maxCmp == 0`, and it opens up other opportunities to optimize rules that don't have `cast`. WDYT?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480831353



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,

Review comment:
       exp: BinaryComparison




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480491875



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max

Review comment:
       `!==` instead of `=!=`? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683410433






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683410287


   **[Test build #128035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128035/testReport)** for PR 29565 at commit [`1a7a3e1`](https://github.com/apache/spark/commit/1a7a3e1f1ae6d87b390035423ebeabca0a163ae9).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479460009



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>

Review comment:
       actually since we plan to handle more types later, this will be replaced. I'll probably do:
   ```scala
   case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) 
     if canImplicitlyCast(fromExp, toType) =>
   ```
   where `canImplicitlyCast` will encapsulate the logic on whether the case is valid.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480824929



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+

Review comment:
       remove this empty line.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479518598



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>

Review comment:
       I was thinking about that. Let's revisit it later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479465939



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>

Review comment:
       Add a comment on why you try to move the cast to left side.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r481413685



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.falseIfNotNull
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case EqualNullSafe(_, _) =>
+          // make sure the expression is evaluated if it is non-deterministic
+          if (exp.deterministic) FalseLiteral else exp

Review comment:
       👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485432133



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {

Review comment:
       updated - let me know if you think this is better




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486063074



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max

Review comment:
       `fromExp <=> max`?
   
   For `cast(fromExp, toType) <=> value`, if `fromExp` is null, original binary expression evaluates to false. But `fromExp == max` evaluates to null, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486858033



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690763193






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485901766



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp

Review comment:
       Do we have a test coverage for this line?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683142673






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487647475



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case EqualNullSafe(_, _) =>
+          EqualNullSafe(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case EqualNullSafe(_, _) =>
+          EqualNullSafe(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+        case _ => exp
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,

Review comment:
       Ya. +1 for `foldable` checking.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485901766



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp

Review comment:
       ~Do we have a test coverage for this line?~ Never mind~




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479701484



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       Thanks. If you are OK I'm slightly inclined to keep the `UnwrapCast` name though since this method is evolving and will expand to more types in follow-ups (should we name it to `UnwrapIntegralAndDateTypeCast` when we added `Date` later?). Plus the comments have already indicated that this only handle integral type at the moment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480498826



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false

Review comment:
       When `exp` is non-deterministic, in Spark, it can have side effect. Naively replaces it by `false` meaning that we skip evaluating`exp`. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690893067






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691373853






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486521031



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)

Review comment:
       Sure will do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485867183



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))

Review comment:
       ```scala
   -    case be @ BinaryComparison(
   -      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
   +    case be @ BinaryComparison(Cast(fromExp, toType: IntegralType, _), Literal(value, _))
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703979



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,

Review comment:
       indentation? Since we have an [official code style guide](https://spark.apache.org/contributing.html), it would save your time if you read.
   ```
   For Scala code, Apache Spark follows the official Scala style guide and Databricks Scala guide. The latter is preferred.
   ```
   - https://github.com/databricks/scala-style-guide




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487646749



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case EqualNullSafe(_, _) =>
+          EqualNullSafe(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case EqualNullSafe(_, _) =>
+          EqualNullSafe(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+        case _ => exp
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,

Review comment:
       indentation nit:
   ```
   def func(
       para1: T,
       para2: T,
       para3: T): R = ...
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479530290



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487155527



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690815772






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703389



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =

Review comment:
       These are referenced in tests so I had to change their scope. Let me know if you have any better idea.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385611






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480784000



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max

Review comment:
       No worries, I never use this implicit operators, `=!=` so it looks very new to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685116009






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479701986



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)

Review comment:
       +1 for @tanelk 's comment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683142261


   **[Test build #128001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128001/testReport)** for PR 29565 at commit [`fc4311b`](https://github.com/apache/spark/commit/fc4311b8f9ef52de1e0ea623ba239a66b03dbd50).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486664847



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)

Review comment:
       Added a comment on the top




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485873421



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))

Review comment:
       In the future, we will add more types. Maybe we can remove the types in the matching and `canImplicitlyCast` in followup PR to make the readability better.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691305169


   **[Test build #128581 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128581/testReport)** for PR 29565 at commit [`5f32ee5`](https://github.com/apache/spark/commit/5f32ee5fda08f4271959589c7f6312195898af08).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703223



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =

Review comment:
       `private[sql]` -> `private[catalyst]`

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, true)`. The if-clause is represented
+   * using `or(isnotnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def trueIfNotNull(e: Expression): Expression =

Review comment:
       `private[sql]` -> `private[catalyst]`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480521584



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       It is hard to answer this question without knowing things like how normally user write queries, how much time this rule uses comparing to other rules, and how much time these specific 2 cases use. It seems the he optimization on these two cases can offer significant speed up, and the extra cost is just the pattern matching:
   
   ```
   else if (maxCmp == 0) {
         exp match {
           case GreaterThan(_, _) =>
             fromExp.falseIfNotNull
           case LessThanOrEqual(_, _) =>
             fromExp.trueIfNotNull
           case LessThan(_, _) =>
             Not(EqualTo(fromExp, Literal(max, fromType)))
           case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
             EqualTo(fromExp, Literal(max, fromType))
           case _ => exp
         }
   ```
   
   which I think should be pretty fast? comparing to the rule itself which may require walking the whole plan tree.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479728328



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =

Review comment:
       Actually I think we can use `private[optimizer]` which is a smaller scope




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479458937



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>

Review comment:
       👍 yup will do this




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479557790



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)

Review comment:
       Is it safe to do the type casting here? We might add more types in `canImplicitlyCast`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690762690


   **[Test build #128538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128538/testReport)** for PR 29565 at commit [`1f87c37`](https://github.com/apache/spark/commit/1f87c37a82bded5c4c9d3e38b7210b9114bba877).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689411721






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479403304



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other

Review comment:
       nit, `case _ => e`

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }

Review comment:
       Since we will add more rules, we will add more conditions for early return. Should we explicitly match the condition that can be simplify? For example,
   
   ```
       case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) if
         fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] && 
         Cast.canUpCast(fromType, toType)) =>
   ```
   Then we start the following code.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]

Review comment:
       I read `IntegralType` as `IntegerType` last night. Must be too sleepy :)

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>

Review comment:
       ```
   case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType)) =>
   ```
   You can do the above that will ensure `toType` will be `IntergralType`. And you don't need to do asInstanceOf laster if you have this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479701531



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       > Also, you may want to mention BinaryComparison like SimplifyBinaryComparison because this optimizer only handles BinaryComparison.
   
   Hmm what do you mean? I did mention binary comparison in the comments, such as:
   ```
   Unwrap casts in binary comparison operations with patterns like following
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690892299


   **[Test build #128539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128539/testReport)** for PR 29565 at commit [`ec88961`](https://github.com/apache/spark/commit/ec88961ff3b48388d8a7847fbdc57d061fc52ff2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480866718



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>

Review comment:
       You mean move `canImplicitlyCast` to `simplifyIntegral` or inline the method in `simplifyIntegral`?
   
   Yeah we could, although later on we may still want to extract these checks into a separate function to improve readability. We can either get rid of `canImplicitlyCast` right now and just replace it with inline checks in `simplifyIntegral`, or keep it and add more checks to it later. I'm fine with either way since this code piece is evolving and will be changed later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703223



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =

Review comment:
       `private[sql]` -> `private`?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, true)`. The if-clause is represented
+   * using `or(isnotnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def trueIfNotNull(e: Expression): Expression =

Review comment:
       `private[sql]` -> `private`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486084222



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  private def getRange(dt: DataType): (Any, Any) = dt match {
+    case ByteType => (Byte.MinValue, Byte.MaxValue)
+    case ShortType => (Short.MinValue, Short.MaxValue)
+    case IntegerType => (Int.MinValue, Int.MaxValue)
+    case LongType => (Long.MinValue, Long.MaxValue)
+  }
+
+  private[optimizer] implicit class ExpressionWrapper(e: Expression) {

Review comment:
       IIRC we should avoid using implicit class, but not very sure, cc @HyukjinKwon @srowen 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480829926



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>

Review comment:
       Are we going to add different logics for different types in `canImplicitlyCast`? It might be hard to maintain. We already ensure `toType` is `IntegralType`, can we just check if `fromExp.dataType is instance of IntegralType`. Thus, we can get rid of `canImplicitlyCast`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478877848



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`

Review comment:
       Yes it is. I think they should be the same type. Will add it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478933473



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp // impossible but safe guard, same below
+        }
+      } else if (maxCmp == 0) {
+        exp match {
+          case GreaterThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(max, fromType)))
+          case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(max, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Compare lower bounds
+      val minCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, minInToType)
+      if (minCmp < 0) {
+        exp match {
+          case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+            return falseIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp
+        }
+      } else if (minCmp == 0) {
+        exp match {
+          case LessThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case GreaterThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(min, fromType)))
+          case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(min, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Now we can assume `value` is within the bound of the source type, e.g., min < value < max
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+
+    case _ => exp
+  }
+
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))

Review comment:
       Will do.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp // impossible but safe guard, same below
+        }
+      } else if (maxCmp == 0) {
+        exp match {
+          case GreaterThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case LessThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(max, fromType)))
+          case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(max, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Compare lower bounds
+      val minCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, minInToType)
+      if (minCmp < 0) {
+        exp match {
+          case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+            return falseIfNotNull(fromExp)
+          case EqualNullSafe(_, _) =>
+            return FalseLiteral
+          case _ => return exp
+        }
+      } else if (minCmp == 0) {
+        exp match {
+          case LessThan(_, _) =>
+            return falseIfNotNull(fromExp)
+          case GreaterThanOrEqual(_, _) =>
+            return trueIfNotNull(fromExp)
+          case GreaterThan(_, _) =>
+            return Not(EqualTo(fromExp, Literal(min, fromType)))
+          case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+            return EqualTo(fromExp, Literal(min, fromType))
+          case _ => return exp
+        }
+      }
+
+      // Now we can assume `value` is within the bound of the source type, e.g., min < value < max
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+
+    case _ => exp
+  }
+
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))
+
+  private[sql] def trueIfNotNull(e: Expression): Expression =
+    Or(IsNotNull(e), Literal(null, BooleanType))

Review comment:
       Will do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486665676



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  private def getRange(dt: DataType): (Any, Any) = dt match {
+    case ByteType => (Byte.MinValue, Byte.MaxValue)
+    case ShortType => (Short.MinValue, Short.MaxValue)
+    case IntegerType => (Int.MinValue, Int.MaxValue)
+    case LongType => (Long.MinValue, Long.MaxValue)
+  }
+
+  private[optimizer] implicit class ExpressionWrapper(e: Expression) {

Review comment:
       Removed the implicit class.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480493279



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       Does it worth to specifically handle `value == max` and `value == min`? Rules will add cost into planning phase, and we probably not going to run into those two cases often in real world queries. Should we handle them as if it's within the range?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685221096






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689577826


   **[Test build #128441 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128441/testReport)** for PR 29565 at commit [`265c169`](https://github.com/apache/spark/commit/265c1698c29730b0dee2865773557d7a3c4c1144).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689579352






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485809871



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {

Review comment:
       Nit, in general, we had better narrow down the parameter from `fromExp` to `fromType`. But, it seems that there is a follow-up plan to extend this optimizer. This looks fine in that case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-684149412


   **[Test build #128128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128128/testReport)** for PR 29565 at commit [`1921601`](https://github.com/apache/spark/commit/1921601d57ea284b418a538b257409bfe6bd5f2e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690893067






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480479800



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)

Review comment:
       ```
    *  - `cast(exp, ty) > value` ==> if(isnotnull(exp), false, null)
   ```
   seems to be more readable.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691373853






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486084839



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       I don't think perf matters that much here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685033827






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487047097



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480894957



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       Those filters are not as effective though. We may add separate rules but that may be more expensive I think (we'll need again to check both sides of the comparison, what types the literal is, the min/max, what kind of expression `fromExp` is, and potential edge cases when both sides have different types etc). 
   
   IMO the advantage of doing it here is that we've already done most of the checks and I don't see much extra cost.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-684149779






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704493



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {

Review comment:
       Just a tip. In general, Apache Spark community recommends not to add to `SQLQuerySuite` because this file is already fat. 
   - For general SQL operation, we prefer SQL file test.
   - For data source only test, we prefer to use more smaller suite. For your test case,  `FileBasedDataSourceSuite` might be better. Please take a look if that is proper to you.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480477785



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType))

Review comment:
       Good point. We are relying on that assumption but it is better to add a check.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682320213






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690762690


   **[Test build #128538 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128538/testReport)** for PR 29565 at commit [`1f87c37`](https://github.com/apache/spark/commit/1f87c37a82bded5c4c9d3e38b7210b9114bba877).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486665937



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       Hmm you mean we can just keep as it is right? I think semantically they are the same?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486827015



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685033112


   **[Test build #128157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128157/testReport)** for PR 29565 at commit [`b5371c5`](https://github.com/apache/spark/commit/b5371c59cf44bf1c0e70571dda7736a4f72bcd23).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479724928



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,

Review comment:
       Will do. I may need to tweak my IDE so don't have to adjust by hand in future. BTW I really hope I can rely on `dev/scalafmt` for this but when I run it it generates lots of irrelevant changes. :(




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486063585



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       can we just convert everything to long and compare?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479727826



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {

Review comment:
       Yes it applies to both v1 and v2. The reason I chose v2 is because v1 `FileSourceScanExec` doesn't expose API to get pushed filters. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487047097



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       This additional check looks good and robust. BTW, do we have a test coverage for this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486827015



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479702717



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, true)`. The if-clause is represented
+   * using `or(isnotnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def trueIfNotNull(e: Expression): Expression =
+    Or(IsNotNull(e), Literal(null, BooleanType))
+
+  private def getRange(ty: DataType): (Any, Any) = ty match {

Review comment:
       nit. Apache Spark usually use `dt: DataType` (rarely, `d: DataType` or `t: DataType`). It would be better to use `dt` here for consistency.
   ```
   $ git grep ' dt: DataType' | wc -l
         31
   $ git grep ' d: DataType' | wc -l
          2
   $ git grep ' t: DataType' | wc -l
          3
   $ git grep 'ty: DataType'
   sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala:  private def getRange(ty: DataType): (Any, Any) = ty match {
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485893094



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparisonSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans.DslLogicalPlan
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.IntegralLiteralTestUtils._
+import org.apache.spark.sql.catalyst.expressions.aggregate.First
+import org.apache.spark.sql.catalyst.optimizer.UnwrapCastInBinaryComparison._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types.{BooleanType, DoubleType, IntegerType}
+
+class UnwrapCastInBinaryComparisonSuite extends PlanTest {
+
+  object Optimize extends RuleExecutor[LogicalPlan] {
+    val batches: List[Batch] =
+      Batch("Unwrap casts in binary comparison", FixedPoint(10),
+        NullPropagation, ConstantFolding, UnwrapCastInBinaryComparison) :: Nil
+  }
+
+  val testRelation: LocalRelation = LocalRelation('a.short, 'b.float)
+
+  test("unwrap casts when literal == max") {
+    val v = Short.MaxValue
+    assertEquivalent('a > v.toInt, 'a.attr.falseIfNotNull)
+    assertEquivalent('a >= v.toInt, 'a === v)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, 'a.attr.trueIfNotNull)
+    assertEquivalent('a < v.toInt, 'a =!= v)
+  }
+
+  test("unwrap casts when literal > max") {
+    val v: Int = positiveInt
+    assertEquivalent('a > v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a >= v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a === v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a <=> v, false)
+    assertEquivalent('a <= v, 'a.attr.trueIfNotNull)
+    assertEquivalent('a < v, 'a.attr.trueIfNotNull)
+  }
+
+  test("unwrap casts when literal == min") {
+    val v = Short.MinValue
+    assertEquivalent('a > v.toInt, 'a =!= v)
+    assertEquivalent('a >= v.toInt, 'a.attr.trueIfNotNull)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, 'a === v)
+    assertEquivalent('a < v.toInt, 'a.attr.falseIfNotNull)
+  }
+
+  test("unwrap casts when literal < min") {
+    val v: Int = negativeInt
+    assertEquivalent('a > v, 'a.attr.trueIfNotNull)
+    assertEquivalent('a >= v, 'a.attr.trueIfNotNull)
+    assertEquivalent('a === v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a <=> v, false)
+    assertEquivalent('a <= v, 'a.attr.falseIfNotNull)
+    assertEquivalent('a < v, 'a.attr.falseIfNotNull)
+  }
+
+  test("unwrap casts when literal is within range (min, max)") {
+    assertEquivalent('a > 300, 'a > 300.toShort)
+    assertEquivalent('a >= 500, 'a >= 500.toShort)
+    assertEquivalent('a === 32766, 'a === 32766.toShort)
+    assertEquivalent('a <=> 32766, 'a <=> 32766.toShort)
+    assertEquivalent('a <= -6000, 'a <= -6000.toShort)
+    assertEquivalent('a < -32767, 'a < -32767.toShort)
+  }
+
+  test("unwrap casts when cast is on rhs") {
+    val v = Short.MaxValue
+    assertEquivalent(Literal(v.toInt) < 'a, 'a.attr.falseIfNotNull)
+    assertEquivalent(Literal(v.toInt) <= 'a, Literal(v) === 'a)
+    assertEquivalent(Literal(v.toInt) === 'a, Literal(v) === 'a)
+    assertEquivalent(Literal(v.toInt) <=> 'a, Literal(v) === 'a)
+    assertEquivalent(Literal(v.toInt) >= 'a, 'a.attr.trueIfNotNull)
+    assertEquivalent(Literal(v.toInt) > 'a, 'a =!= v)
+
+    assertEquivalent(Literal(30) <= 'a, Literal(30.toShort) <= 'a)
+  }
+
+  test("unwrap cast should have no effect when input is not integral type") {
+    assertEquivalent('b > 42.0, Cast('b, DoubleType) > 42.0)
+    assertEquivalent('b >= 42.0, Cast('b, DoubleType) >= 42.0)
+    assertEquivalent('b === 42.0, Cast('b, DoubleType) === 42.0)
+    assertEquivalent('b <=> 42.0, Cast('b, DoubleType) <=> 42.0)
+    assertEquivalent('b <= 42.0, Cast('b, DoubleType) <= 42.0)
+    assertEquivalent('b < 42.0, Cast('b, DoubleType) < 42.0)
+    assertEquivalent(Literal(42.0) > 'b, Literal(42.0) > Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) >= 'b, Literal(42.0) >= Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) === 'b, Literal(42.0) === Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) <=> 'b, Literal(42.0) <=> Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) <= 'b, Literal(42.0) <= Cast('b, DoubleType))
+    assertEquivalent(Literal(42.0) < 'b, Literal(42.0) < Cast('b, DoubleType))
+  }
+
+  test("unwrap cast should skip when expression is non-deterministic") {
+    assertEquivalent(Cast(First('a, ignoreNulls = true), IntegerType) <=> positiveInt,
+      Cast(First('a, ignoreNulls = true), IntegerType) <=> positiveInt)
+    assertEquivalent(Cast(First('a, ignoreNulls = true), IntegerType) <=> negativeInt,
+      Cast(First('a, ignoreNulls = true), IntegerType) <=> negativeInt)

Review comment:
       The following will show the test intention more clearly.
   ```scala
   -    assertEquivalent(Cast(First('a, ignoreNulls = true), IntegerType) <=> positiveInt,
   -      Cast(First('a, ignoreNulls = true), IntegerType) <=> positiveInt)
   -    assertEquivalent(Cast(First('a, ignoreNulls = true), IntegerType) <=> negativeInt,
   -      Cast(First('a, ignoreNulls = true), IntegerType) <=> negativeInt)
   +    Seq(positiveInt, negativeInt).foreach { v =>
   +      val e = Cast(First('a, ignoreNulls = true), IntegerType) <=> v
   +      assertEquivalent(e, e)
   +    }
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385050


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128032/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385045






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703682



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastSuite.scala
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans.DslLogicalPlan
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types.DoubleType
+
+class UnwrapCastSuite extends PlanTest {
+
+  object Optimize extends RuleExecutor[LogicalPlan] {
+    val batches: List[Batch] =
+      Batch("Unwrap casts", FixedPoint(10), ConstantFolding, UnwrapCast) :: Nil
+  }
+
+  val testRelation: LocalRelation = LocalRelation('a.short, 'b.float)
+
+  test("unwrap casts when literal == max") {
+    val v = Short.MaxValue
+    assertEquivalent('a > v.toInt, UnwrapCast.falseIfNotNull('a))
+    assertEquivalent('a >= v.toInt, 'a === v)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, UnwrapCast.trueIfNotNull('a))
+    assertEquivalent('a < v.toInt, 'a =!= v)
+  }
+
+  test("unwrap casts when literal > max") {
+    val v: Int = Short.MaxValue + 100
+    assertEquivalent('a > v, UnwrapCast.falseIfNotNull('a))
+    assertEquivalent('a >= v, UnwrapCast.falseIfNotNull('a))
+    assertEquivalent('a === v, UnwrapCast.falseIfNotNull('a))
+    assertEquivalent('a <=> v, false)
+    assertEquivalent('a <= v, UnwrapCast.trueIfNotNull('a))
+    assertEquivalent('a < v, UnwrapCast.trueIfNotNull('a))
+  }
+
+  test("unwrap casts when literal == min") {
+    val v = Short.MinValue
+    assertEquivalent('a > v.toInt, 'a =!= v)
+    assertEquivalent('a >= v.toInt, UnwrapCast.trueIfNotNull('a))
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, 'a === v)
+    assertEquivalent('a < v.toInt, UnwrapCast.falseIfNotNull('a))
+  }
+
+  test("unwrap casts when literal < min") {
+    val v: Int = Short.MinValue - 100

Review comment:
       ```scala
   -    val v: Int = Short.MinValue - 100
   +    val v: Int = negativeInt
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689415160


   **[Test build #128441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128441/testReport)** for PR 29565 at commit [`265c169`](https://github.com/apache/spark/commit/265c1698c29730b0dee2865773557d7a3c4c1144).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690816027


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128538/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690802886


   **[Test build #128539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128539/testReport)** for PR 29565 at commit [`ec88961`](https://github.com/apache/spark/commit/ec88961ff3b48388d8a7847fbdc57d061fc52ff2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385029






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480473222



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType))

Review comment:
       Is it possible that in `Cast(fromExp, toType1, _)` and `Literal(value, toType2: IntegralType)`, `toType1` and `toType2` are different? Should we check? They will be eventually the same by other rules, but we want this rule kick in after they are the same type.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704215



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {

Review comment:
       @sunchao . BTW, why do we test `DataSourceV2` only? Your PR is neutral to V1 or V2, isn't it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487155527



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485348560



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))

Review comment:
       Since we are not really casting the Literal, how about 
   ```
         Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
   
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479463424



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>

Review comment:
       `case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) if maxCmp > 0 ` might be more readable, and you can combine the next pattern matching.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690801318






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683383336






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685219980


   **[Test build #128165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128165/testReport)** for PR 29565 at commit [`aec3053`](https://github.com/apache/spark/commit/aec3053103a0d9448769722131abd046ec4ac20b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480860549



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>

Review comment:
       👍 yes we should skip if the input is not qualified




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690893067






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486044404



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)

Review comment:
       ah I see, we can't support it, as `cast(long_col as int)` may overflow and we can't move cast to the other side. Can we add a comment here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479723920



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       How about `UnwrapCastInBinaryComparison`? 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =

Review comment:
       👍 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480819992



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>

Review comment:
        `   case BinaryComparison(Literal(_, dt1), Cast(_, dt2, _)) if dt1.sameType(dt2) =>` to reject cases that can not be optimized.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683383336






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486827015



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       semantically they are the same, but the code is easier to read if we do `if (value > max) ...` instead of `(minCmp.signum, maxCmp.signum, exp) match ...` 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479725043



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {

Review comment:
       Okay. I think `FileBasedDataSourceSuite` may be a better place. Let me move these to there.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691617153


   Thank you all for the review and commit!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485876275



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  private def getRange(dt: DataType): (Any, Any) = dt match {
+    case ByteType => (Byte.MinValue, Byte.MaxValue)
+    case ShortType => (Short.MinValue, Short.MaxValue)
+    case IntegerType => (Int.MinValue, Int.MaxValue)
+    case LongType => (Long.MinValue, Long.MaxValue)

Review comment:
       We probably need to handle this exception in the caller to prevent the job being killed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479460009



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>

Review comment:
       actually since we plan to handle more types later, this will be replaced. I'll probably do:
   ```scala
   case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) if canImplicitlyCast(fromExp, toType) =>
   ```
   where `canImplicitlyCast` will encapsulate the logic on whether the case is valid.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690815772


   **[Test build #128538 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128538/testReport)** for PR 29565 at commit [`1f87c37`](https://github.com/apache/spark/commit/1f87c37a82bded5c4c9d3e38b7210b9114bba877).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689415160


   **[Test build #128441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128441/testReport)** for PR 29565 at commit [`265c169`](https://github.com/apache/spark/commit/265c1698c29730b0dee2865773557d7a3c4c1144).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682249127


   **[Test build #127969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127969/testReport)** for PR 29565 at commit [`a39b99a`](https://github.com/apache/spark/commit/a39b99ad8673dd82a862a081d303b30cdc9a862d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685221096






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479672380



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)

Review comment:
       nit: missing safe guard case




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690802886


   **[Test build #128539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128539/testReport)** for PR 29565 at commit [`ec88961`](https://github.com/apache/spark/commit/ec88961ff3b48388d8a7847fbdc57d061fc52ff2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690816017


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487646973



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case EqualNullSafe(_, _) =>
+          EqualNullSafe(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        // make sure the expression is evaluated if it is non-deterministic
+        case EqualNullSafe(_, _) if exp.deterministic =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case EqualNullSafe(_, _) =>
+          EqualNullSafe(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+        case _ => exp
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,

Review comment:
       BTW I'll also check `!from.foldable`, otherwise it's simpler to not run this rule.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683142673






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487155527



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       thanks - let me switch to `if else` for better readability here.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691373497


   **[Test build #128581 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128581/testReport)** for PR 29565 at commit [`5f32ee5`](https://github.com/apache/spark/commit/5f32ee5fda08f4271959589c7f6312195898af08).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480547347



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:

Review comment:
       I don't see much difference between the two and I think the expression itself should be easy to understand. With that said I don't hold strong opinion on this and either way is OK to me :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-684506726


   **[Test build #128128 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128128/testReport)** for PR 29565 at commit [`1921601`](https://github.com/apache/spark/commit/1921601d57ea284b418a538b257409bfe6bd5f2e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690762690






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486084726



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       In CBO, we convert everything to `BigDecimal` and compare, which simplifies the code a lot.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486511586



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  private def getRange(dt: DataType): (Any, Any) = dt match {
+    case ByteType => (Byte.MinValue, Byte.MaxValue)
+    case ShortType => (Short.MinValue, Short.MaxValue)
+    case IntegerType => (Int.MinValue, Int.MaxValue)
+    case LongType => (Long.MinValue, Long.MaxValue)

Review comment:
       Will do. Ideally this should never happen though unless someone adds another subclass under `IntegralType` (and in that case the implementor should fulfill their responsibility by checking every place that does pattern matching on the parent type). It's unfortunate that we can't seal the class and rely on Scala's exhaustive pattern match compile-time check :(
   
   Regarding catching the exception, I don't think it is necessary since it should be an error in query compilation before job starts. I see similar pattern in [ColumnType](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala#L767).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690801318






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479705394



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       The above comments are the same purpose. `UnwrapCast` is too broad and description is not enough always.
   - `SimplifyBinaryComparison` has a clear meaning to the audience that it works only for `BinaryComparision`. Your optimize also does.
   
   I don't think we can enumerate all types into the optimizer name like `UnwrapIntegralAndDateTypeCast`. We can choose more proper name at that context. The current `UnwrapCast` looks like a little over-claim to me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478935572



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return falseIfNotNull(fromExp)

Review comment:
       Here we need an early return - without the `return` keyword the execution will continue and treat this as the case when `value` is within range `(min, max)`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690892299






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704999



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =

Review comment:
       You can use `private[catalyst]`. I updated my comment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480476764



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:

Review comment:
       ```
   if `exp < max of exp type > value`, the cases are of following:
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485906677



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))

Review comment:
       It's because it's not redundant. `canImplicitlyCast` already checks it.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))

Review comment:
       It's because it's redundant. `canImplicitlyCast` already checks it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682319626


   **[Test build #127969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127969/testReport)** for PR 29565 at commit [`a39b99a`](https://github.com/apache/spark/commit/a39b99ad8673dd82a862a081d303b30cdc9a862d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704859



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+        withTempPath { dir =>
+          spark.range(100).map(i => (i.toShort, i.toString)).toDF("id", "s")
+            .write
+            .format(format)
+            .save(dir.getCanonicalPath)
+          val df = spark.read.format(format).load(dir.getCanonicalPath)
+
+          // cases when value == MAX
+          var v = Short.MaxValue
+          checkPushedFilters(
+            format,
+            df.where('id > v.toInt),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id >= v.toInt),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where('id === v.toInt),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where('id <=> v.toInt),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where('id <= v.toInt),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where('id < v.toInt),
+            Array(sources.IsNotNull("id"), sources.Not(sources.EqualTo("id", v))))
+
+          // cases when value > MAX
+          var v1: Int = Short.MaxValue.toInt + 100
+          checkPushedFilters(
+            format,
+            df.where('id > v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id >= v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id === v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id <=> v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id <= v1),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where('id < v1),
+            Array(sources.IsNotNull("id")))
+
+          // cases when value = MIN
+          v = Short.MinValue
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) < 'id),
+            Array(sources.IsNotNull("id"), sources.Not(sources.EqualTo("id", v))))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) <= 'id),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) === 'id),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) >= 'id),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) > 'id),
+            Array(),
+            noScan = true)
+
+          // cases when value < MIN
+          v1 = Short.MinValue.toInt - 100
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) < 'id),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) <= 'id),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) === 'id),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) >= 'id),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) > 'id),
+            Array(),
+            noScan = true)

Review comment:
       Shall we minimize the above 20 lines into 5 lines like the following? The above looks too verbose unnecessarily.
   ```scala
   checkPushedFilters(format, df.where(lit(v1) < 'id), Array(sources.IsNotNull("id")))
   checkPushedFilters(format, df.where(lit(v1) <= 'id), Array(sources.IsNotNull("id")))
   checkPushedFilters(format, df.where(lit(v1) === 'id), Array(), noScan = true)
   checkPushedFilters(format, df.where(lit(v1) >= 'id), Array(), noScan = true)
   checkPushedFilters(format, df.where(lit(v1) > 'id), Array(), noScan = true)
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478936990



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]

Review comment:
       It is not - can be other types. Why do you think it may always be `Int`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479590420



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)

Review comment:
       Yes currently it's safe since we only allow integral types. Later when we add more types we'll add safe guard here.
   
   We can do the matching on `IntegralType` but I think it might be slightly better to encapsulate the logic in `canImplicitylyCast` so that we can expand it later such as casting from long to double with digits checking etc. But yeah I don't have strong preference on either at this moment. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479593866



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)

Review comment:
       Or we can keep the existing and add the above `toType: IntegralType`, then we can remove the casting.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480497366



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType))

Review comment:
       Yeah, without checking, we might break the query plan if the plan we get here is before type coercion.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486858033



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]

Review comment:
       This was my bad :) I suggested to use pattern matching instead of mixing `if else` and pattern matching.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485895878



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp

Review comment:
       Maybe,
   ```scala
   -      case (_, _, _) => exp
   +      case _ => exp
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691208083






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691305654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480525850



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max

Review comment:
       I borrowed implicit operators on expressions from [here](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala#L92). Is `!==` a better known inequality operator in Spark?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] tanelk commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683320988


   I wondered how it handles null literal, might want to add a test case like this:
   
   ```
     test("unwrap casts when literal is null") {
       val intLit = Literal.create(null, IntegerType)
       val shortLit = Literal.create(null, ShortType)
       assertEquivalent('a > intLit, 'a > shortLit)
       assertEquivalent('a >= intLit, 'a >= shortLit)
       assertEquivalent('a === intLit, 'a === shortLit)
       assertEquivalent('a <=> intLit, 'a <=> shortLit)
       assertEquivalent('a <= intLit, 'a <= shortLit)
       assertEquivalent('a < intLit, 'a < shortLit)
     }
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479700098



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       Shall we rename `UnwrapCast ` to more precise one like `UnwrapIntegralCast` because this optimizer only handle `IntegralType`? We can use `UnwrapCast` later when we support more types like `FractionalType`, `Date`, `String` and so on.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480539363



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)

Review comment:
       Yes `toType`. As mentioned below, I can make the terms consistent and change to `cast(fromExp, toType)` 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683142261


   **[Test build #128001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128001/testReport)** for PR 29565 at commit [`fc4311b`](https://github.com/apache/spark/commit/fc4311b8f9ef52de1e0ea623ba239a66b03dbd50).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485374894



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       Let's consider it in the followup PR.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683211038






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480476290



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)

Review comment:
       `ty` to `dt`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486044204



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)

Review comment:
       For `cast(long_col as int) = int_literal`, we don't support it?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479530214



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }

Review comment:
       I think predicate pushdown is the main use case but this can also help to simplify the binary comparisons. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683352883


   > I wondered how it handles null literal, might want to add a test case like this:
   > 
   > ```
   >   test("unwrap casts when literal is null") {
   >     val intLit = Literal.create(null, IntegerType)
   >     val shortLit = Literal.create(null, ShortType)
   >     assertEquivalent('a > intLit, 'a > shortLit)
   >     assertEquivalent('a >= intLit, 'a >= shortLit)
   >     assertEquivalent('a === intLit, 'a === shortLit)
   >     assertEquivalent('a <=> intLit, 'a <=> shortLit)
   >     assertEquivalent('a <= intLit, 'a <= shortLit)
   >     assertEquivalent('a < intLit, 'a < shortLit)
   >   }
   > ```
   
   Sure. I can add tests for this. But notice that nulls should have already been handled by `NullPropagation` though so the binary comparisons should have been transformed to `null` or something else.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683211038






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479559320



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)

Review comment:
       If we match `    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType))` it's safer and no need for casting.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486496950



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {

Review comment:
       Yes this method is evolving and will be enhanced in the follow-ups.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479466938



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))

Review comment:
       Do we have a test that if `swap(exp)` is unable to unwrap cast, we will swap it back to the original form?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun closed pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29565:
URL: https://github.com/apache/spark/pull/29565


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485867183



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))

Review comment:
       In this case, we cannot use `DataType` instead of `IntegralType`. So, I updated partially.
   ```scala
   -    case be @ BinaryComparison(
   -      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
   +    case be @ BinaryComparison(Cast(fromExp, toType: IntegralType, _), Literal(value, _))
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704933



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {
+        withTempPath { dir =>
+          spark.range(100).map(i => (i.toShort, i.toString)).toDF("id", "s")
+            .write
+            .format(format)
+            .save(dir.getCanonicalPath)
+          val df = spark.read.format(format).load(dir.getCanonicalPath)
+
+          // cases when value == MAX
+          var v = Short.MaxValue
+          checkPushedFilters(
+            format,
+            df.where('id > v.toInt),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id >= v.toInt),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where('id === v.toInt),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where('id <=> v.toInt),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where('id <= v.toInt),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where('id < v.toInt),
+            Array(sources.IsNotNull("id"), sources.Not(sources.EqualTo("id", v))))
+
+          // cases when value > MAX
+          var v1: Int = Short.MaxValue.toInt + 100
+          checkPushedFilters(
+            format,
+            df.where('id > v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id >= v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id === v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id <=> v1),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where('id <= v1),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where('id < v1),
+            Array(sources.IsNotNull("id")))
+
+          // cases when value = MIN
+          v = Short.MinValue
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) < 'id),
+            Array(sources.IsNotNull("id"), sources.Not(sources.EqualTo("id", v))))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) <= 'id),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) === 'id),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) >= 'id),
+            Array(sources.IsNotNull("id"), sources.EqualTo("id", v)))
+          checkPushedFilters(
+            format,
+            df.where(lit(v.toInt) > 'id),
+            Array(),
+            noScan = true)
+
+          // cases when value < MIN
+          v1 = Short.MinValue.toInt - 100
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) < 'id),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) <= 'id),
+            Array(sources.IsNotNull("id")))
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) === 'id),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) >= 'id),
+            Array(),
+            noScan = true)
+          checkPushedFilters(
+            format,
+            df.where(lit(v1) > 'id),
+            Array(),
+            noScan = true)

Review comment:
       For the other test cases, it's the same. We don't need to break lines when you make it one liner.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683383249






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682249490






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479525498



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))

Review comment:
       Good idea. I'll add a test for this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479726261



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)

Review comment:
       Sure I'll do that.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480476764



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:

Review comment:
       if `exp < max of exp type < value`, the cases are of following:
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-684149412


   **[Test build #128128 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128128/testReport)** for PR 29565 at commit [`1921601`](https://github.com/apache/spark/commit/1921601d57ea284b418a538b257409bfe6bd5f2e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704560



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {

Review comment:
       We recommend not to use irrelevant SQLConfs in the test case if the PR is neutral. If this is the test case limitation at 3601 ~ 3604, we had better change it to test with the default configuration.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385045


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691305654






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479729572



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala
##########
@@ -881,6 +883,90 @@ class FileBasedDataSourceSuite extends QueryTest
       }
     }
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+        format: String,
+        df: DataFrame,
+        filters: Array[sources.Filter],
+        noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan.find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {

Review comment:
       @dongjoon-hyun I kept this since `FileSourceScanExec` in v1 doesn't expose API to get pushed filters. Let me know if you have better idea to test this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385518


   **[Test build #128035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128035/testReport)** for PR 29565 at commit [`1a7a3e1`](https://github.com/apache/spark/commit/1a7a3e1f1ae6d87b390035423ebeabca0a163ae9).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682249490






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685033827






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-682320213






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685172317






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r481330767



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.falseIfNotNull
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case EqualNullSafe(_, _) =>
+          // make sure the expression is evaluated if it is non-deterministic
+          if (exp.deterministic) FalseLiteral else exp

Review comment:
       move the if into case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480858717



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.falseIfNotNull
+          // falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case EqualNullSafe(_, _) =>
+          // make sure the expression is evaluated if it is non-deterministic
+          if (exp.deterministic) FalseLiteral else exp
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          fromExp.falseIfNotNull
+        case LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          fromExp.falseIfNotNull
+        case EqualNullSafe(_, _) =>
+          // make sure the expression is evaluated if it is non-deterministic
+          if (exp.deterministic) FalseLiteral else exp
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          fromExp.falseIfNotNull
+        case GreaterThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+        case _ => exp
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+    Cast.canUpCast(fromExp.dataType, toType)

Review comment:
       No, it is not required but just to make the method self-complete. 
   
   > For Cast.canUpCast, seems this will always be true since it's already in the cast expression constructed by Spark
   
   Hmm I'm not sure. Users can say `cast(<some_long_column> as int) > 42` and we'd want to skip this case, I think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703283



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case LessThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          falseIfNotNull(fromExp)
+        case EqualNullSafe(_, _) =>
+          FalseLiteral
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          falseIfNotNull(fromExp)
+        case GreaterThanOrEqual(_, _) =>
+          trueIfNotNull(fromExp)
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, false)`. The if-clause is represented
+   * using `and(isnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def falseIfNotNull(e: Expression): Expression =
+    And(IsNull(e), Literal(null, BooleanType))
+
+  /**
+   * Wraps input expression `e` with `if(isnull(e), null, true)`. The if-clause is represented
+   * using `or(isnotnull(e), null)` which is semantically equivalent by applying 3-valued logic.
+   */
+  private[sql] def trueIfNotNull(e: Expression): Expression =
+    Or(IsNotNull(e), Literal(null, BooleanType))
+
+  private def getRange(ty: DataType): (Any, Any) = ty match {

Review comment:
       Good to know. Will change. Thanks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480479864



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false

Review comment:
       Hmm. I think even if `exp` is non-deterministic, evaluating `cast(exp, ty)` will generate a value of type `ty` which will be either:
   - a valid value within the range of `ty` and thus will not equal to `value` which is > max of `ty`
   - null which means the result of the comparison will become false since `value` is not null.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683385402


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480473760



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType: IntegralType))

Review comment:
       Maybe just do 
   ```
   BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
   ```
   to make sure, and it will be readable as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486524454



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)

Review comment:
       Sure will add comments about this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485373288



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       Yes +1 on exploring that. I think we should keep the `maxCmp == 0` case for now though before that is ready, since this is more optimized with not much extra cost (without the `maxCmp == 0` we still have to do pattern matching in the else case).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479700376



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {

Review comment:
       You can keep the private function name `private def unwrapCast`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683210663


   **[Test build #128001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128001/testReport)** for PR 29565 at commit [`fc4311b`](https://github.com/apache/spark/commit/fc4311b8f9ef52de1e0ea623ba239a66b03dbd50).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480823284



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.

Review comment:
       In Spark, we typically have comment above the `case`.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.

Review comment:
       ditto.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486666115



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max

Review comment:
       Fixed this and added tests to cover it. Please take a look.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689411721






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480828168



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.falseIfNotNull
+          // falseIfNotNull(fromExp)
+        case LessThan(_, _) | LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case EqualNullSafe(_, _) =>
+          // make sure the expression is evaluated if it is non-deterministic
+          if (exp.deterministic) FalseLiteral else exp
+        case _ => exp // impossible but safe guard, same below
+      }
+    } else if (maxCmp == 0) {
+      exp match {
+        case GreaterThan(_, _) =>
+          fromExp.falseIfNotNull
+        case LessThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case LessThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(max, fromType)))
+        case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(max, fromType))
+        case _ => exp
+      }
+    } else if (minCmp < 0) {
+      exp match {
+        case GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _) =>
+          fromExp.falseIfNotNull
+        case EqualNullSafe(_, _) =>
+          // make sure the expression is evaluated if it is non-deterministic
+          if (exp.deterministic) FalseLiteral else exp
+        case _ => exp
+      }
+    } else if (minCmp == 0) {
+      exp match {
+        case LessThan(_, _) =>
+          fromExp.falseIfNotNull
+        case GreaterThanOrEqual(_, _) =>
+          fromExp.trueIfNotNull
+        case GreaterThan(_, _) =>
+          Not(EqualTo(fromExp, Literal(min, fromType)))
+        case LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
+          EqualTo(fromExp, Literal(min, fromType))
+        case _ => exp
+      }
+    } else {
+      // This means `value` is within range `(min, max)`. Optimize this by moving the cast to the
+      // literal side.
+      val lit = Cast(Literal(value), fromType)
+      exp match {
+        case GreaterThan(_, _) => GreaterThan(fromExp, lit)
+        case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
+        case EqualTo(_, _) => EqualTo(fromExp, lit)
+        case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
+        case LessThan(_, _) => LessThan(fromExp, lit)
+        case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
+        case _ => exp
+      }
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+    Cast.canUpCast(fromExp.dataType, toType)

Review comment:
       Is this check still required? We check it in the pattern matching for `toType.isInstanceOf[IntegralType]`. For Cast.canUpCast, seems this will always be true since it's already in the cast expression constructed by Spark, right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683383249


   **[Test build #128032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128032/testReport)** for PR 29565 at commit [`c813291`](https://github.com/apache/spark/commit/c8132915b70200ddc0acd5202493cb261ebe2b28).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690893067






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r487156064



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where:
+ *   1). `fromType` (of `fromExp`) and `toType` are of integral types (i.e., byte, short, int and
+ *     long)
+ *   2). `fromType` can be safely coerced to `toType` without precision loss (e.g., short to int,
+ *     int to long, but not long to int)
+ *
+ * If the above conditions are satisfied, the rule checks to see if the literal `value` is within
+ * range `(min, max)`, where `min` and `max` are the minimum and maximum value of `fromType`,
+ * respectively. If this is true then it means we can safely cast `value` to `fromType` and thus
+ * able to move the cast to the literal side. That is:
+ *
+ *   `cast(fromExp, toType) op value` ==> `fromExp op cast(value, fromType)`
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <=> value` ==> cast(fromExp, toType) <=> value (if `fromExp` is
+ *       non-deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp <=> max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, literalType), Cast(fromExp, toType, _))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, literalType))
+        if canImplicitlyCast(fromExp, toType, literalType) =>
+      simplifyIntegralComparison(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegralComparison(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp
+
+      case (_, 0, GreaterThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        falseIfNotNull(fromExp)
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        falseIfNotNull(fromExp)
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        trueIfNotNull(fromExp)
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, EqualNullSafe(_, _)) =>
+        EqualNullSafe(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case _ => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType,
+      literalType: DataType): Boolean = {
+    toType.sameType(literalType) &&

Review comment:
       I tried to come up with a test for this but it seems the query compiler always wrap a cast to make sure type from both sides are the same.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-684508014






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480858509



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _: IntegralType, _), Literal(value, toType: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>

Review comment:
       maybe we can check in `simplifyIntegral`. If doesn't pass the check, just return `exp`. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-691334731






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486520527



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below

Review comment:
       You're right. This comment is stale. Will remove.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479558357



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(exp), null, false)` is represented using conjunction
+ * `and(isnull(exp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(exp), null, true)` is represented with `or(isnotnull(exp), null)`.
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      // Not a canonical form. In this case we first canonicalize the expression by swapping the
+      // literal and cast side, then process the result and swap the literal and cast again to
+      // restore the original order.
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType))
+      if canImplicitlyCast(fromExp, toType) =>
+
+      // In case both sides have integral type, optimize the comparison by removing casts or
+      // moving cast to the literal side.
+      simplifyIntegral(exp, fromExp, toType.asInstanceOf[IntegralType], value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: Expression,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+
+    if (maxCmp > 0) {
+      exp match {
+        case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+          falseIfNotNull(fromExp)

Review comment:
       nit, you might consider to add an implicit conversion rule in this class, so you can directly call `fromExp.falseIfNotNull`. Might be a little bit more readable. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486544561



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max

Review comment:
       I think you are right. Good catch! Let me see how to add a test for this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478898433



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`

Review comment:
       Okay I revised the comments. Please take another look.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690802886






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r486649299



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  private def getRange(dt: DataType): (Any, Any) = dt match {
+    case ByteType => (Byte.MinValue, Byte.MaxValue)
+    case ShortType => (Short.MinValue, Short.MaxValue)
+    case IntegerType => (Int.MinValue, Int.MaxValue)
+    case LongType => (Long.MinValue, Long.MaxValue)
+  }
+
+  private[optimizer] implicit class ExpressionWrapper(e: Expression) {

Review comment:
       Sure. I can switch to use simple methods. I think it doesn't matter much here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-689579352






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690893067






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479458556



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }

Review comment:
       Sure will do that.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479704560



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
##########
@@ -3582,6 +3582,163 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
     checkAnswer(sql("SELECT 0 FROM ( SELECT * FROM B JOIN C USING (id)) " +
       "JOIN ( SELECT * FROM B JOIN C USING (id)) USING (id)"), Row(0))
   }
+
+  test("test casts pushdown on orc/parquet for integral types") {
+    def checkPushedFilters(
+      format: String,
+      df: DataFrame,
+      filters: Array[sources.Filter],
+      noScan: Boolean = false): Unit = {
+      val scanExec = df.queryExecution.sparkPlan
+        .find(_.isInstanceOf[BatchScanExec])
+      if (noScan) {
+        assert(scanExec.isEmpty)
+        return
+      }
+      val scan = scanExec.get.asInstanceOf[BatchScanExec].scan
+      format match {
+        case "orc" =>
+          assert(scan.isInstanceOf[OrcScan])
+          assert(scan.asInstanceOf[OrcScan].pushedFilters === filters)
+        case "parquet" =>
+          assert(scan.isInstanceOf[ParquetScan])
+          assert(scan.asInstanceOf[ParquetScan].pushedFilters === filters)
+        case _ =>
+          fail(s"unknown format $format")
+      }
+    }
+
+    Seq("orc", "parquet").foreach { format =>
+      withSQLConf(SQLConf.USE_V1_SOURCE_LIST.key -> "") {

Review comment:
       We recommend not to use irrelevant SQLConfs in the test case if the PR is neutral.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-685116009






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690816017






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480494704



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:

Review comment:
       Yeah, it's clear. Should we make it `if(isnotnull(fromExp), false, null)` instead?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-684508014






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690816017






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r485812296



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) === value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) <=> value` ==> false (only if `fromExp` is deterministic)
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> if(isnull(fromExp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(fromExp, toType) > value` ==> if(isnull(fromExp), null, false)
+ *  - `cast(fromExp, toType) >= value` ==> fromExp == max
+ *  - `cast(fromExp, toType) === value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <=> value` ==> fromExp == max
+ *  - `cast(fromExp, toType) <= value` ==> if(isnull(fromExp), null, true)
+ *  - `cast(fromExp, toType) < value` ==> fromExp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.
+ *
+ * Further, the above `if(isnull(fromExp), null, false)` is represented using conjunction
+ * `and(isnull(fromExp), null)`, to enable further optimization and filter pushdown to data sources.
+ * Similarly, `if(isnull(fromExp), null, true)` is represented with `or(isnotnull(fromExp), null)`.
+ */
+object UnwrapCastInBinaryComparison extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan =>
+      l transformExpressionsUp {
+        case e @ BinaryComparison(_, _) => unwrapCast(e)
+      }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    // Not a canonical form. In this case we first canonicalize the expression by swapping the
+    // literal and cast side, then process the result and swap the literal and cast again to
+    // restore the original order.
+    case BinaryComparison(Literal(_, toType: IntegralType), Cast(fromExp, _: IntegralType, _))
+        if canImplicitlyCast(fromExp, toType) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case _ => e
+      }
+      swap(unwrapCast(swap(exp)))
+
+    // In case both sides have integral type, optimize the comparison by removing casts or
+    // moving cast to the literal side.
+    case be @ BinaryComparison(
+      Cast(fromExp, toType: IntegralType, _), Literal(value, _: IntegralType))
+        if canImplicitlyCast(fromExp, toType) =>
+      simplifyIntegral(be, fromExp, toType, value)
+
+    case _ => exp
+  }
+
+  /**
+   * Check if the input `value` is within range `(min, max)` of the `fromType`, where `min` and
+   * `max` are the minimum and maximum value of the `fromType`. If the above is true, this
+   * optimizes the expression by moving the cast to the literal side. Otherwise if result is not
+   * true, this replaces the input binary comparison `exp` with simpler expressions.
+   */
+  private def simplifyIntegral(
+      exp: BinaryComparison,
+      fromExp: Expression,
+      toType: IntegralType,
+      value: Any): Expression = {
+
+    val fromType = fromExp.dataType
+    val (min, max) = getRange(fromType)
+    val (minInToType, maxInToType) = {
+      (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+    }
+    val ordering = toType.ordering.asInstanceOf[Ordering[Any]]
+    val minCmp = ordering.compare(value, minInToType)
+    val maxCmp = ordering.compare(value, maxInToType)
+    val lit = Cast(Literal(value), fromType)
+
+    (minCmp.signum, maxCmp.signum, exp) match {
+      case (_, 1, EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 1, LessThan(_, _) | LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (_, 1, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (_, 1, _) => exp // impossible but safe guard, same below
+
+      case (_, 0, GreaterThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (_, 0, LessThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (_, 0, LessThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(max, fromType)))
+      case (_, 0, GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(max, fromType))
+      case (_, 0, _) => exp
+
+      case (-1, _, GreaterThan(_, _) | GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (-1, _, LessThan(_, _) | LessThanOrEqual(_, _) | EqualTo(_, _)) =>
+        fromExp.falseIfNotNull
+      // make sure the expression is evaluated if it is non-deterministic
+      case (-1, _, EqualNullSafe(_, _)) if exp.deterministic =>
+        FalseLiteral
+      case (-1, _, _) => exp
+
+      case (0, _, LessThan(_, _)) =>
+        fromExp.falseIfNotNull
+      case (0, _, GreaterThanOrEqual(_, _)) =>
+        fromExp.trueIfNotNull
+      case (0, _, GreaterThan(_, _)) =>
+        Not(EqualTo(fromExp, Literal(min, fromType)))
+      case (0, _, LessThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _)) =>
+        EqualTo(fromExp, Literal(min, fromType))
+      case (0, _, _) => exp
+
+      case (_, _, GreaterThan(_, _)) => GreaterThan(fromExp, lit)
+      case (_, _, GreaterThanOrEqual(_, _)) => GreaterThanOrEqual(fromExp, lit)
+      case (_, _, EqualTo(_, _)) => EqualTo(fromExp, lit)
+      case (_, _, EqualNullSafe(_, _)) => EqualNullSafe(fromExp, lit)
+      case (_, _, LessThan(_, _)) => LessThan(fromExp, lit)
+      case (_, _, LessThanOrEqual(_, _)) => LessThanOrEqual(fromExp, lit)
+      case (_, _, _) => exp
+    }
+  }
+
+  /**
+   * Check if the input `fromExp` can be safely cast to `toType` without any loss of precision,
+   * i.e., the conversion is injective. Note this only handles the case when both sides are of
+   * integral type.
+   */
+  private def canImplicitlyCast(fromExp: Expression, toType: DataType): Boolean = {
+    fromExp.dataType.isInstanceOf[IntegralType] && toType.isInstanceOf[IntegralType] &&
+      Cast.canUpCast(fromExp.dataType, toType)
+  }
+
+  private def getRange(dt: DataType): (Any, Any) = dt match {
+    case ByteType => (Byte.MinValue, Byte.MaxValue)
+    case ShortType => (Short.MinValue, Short.MaxValue)
+    case IntegerType => (Int.MinValue, Int.MaxValue)
+    case LongType => (Long.MinValue, Long.MaxValue)

Review comment:
       You may want to safe guard here; `case _: throw new UnsupportedOperationException`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480521584



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       It is hard to answer this question without knowing things like how normally user write queries, how much time this rule uses comparing to other rules, and how much time these specific 2 cases use. IMO the optimization on these two cases can offer significant speed up, and the extra cost is just the pattern matching:
   
   ```
   else if (maxCmp == 0) {
         exp match {
           case GreaterThan(_, _) =>
             fromExp.falseIfNotNull
           case LessThanOrEqual(_, _) =>
             fromExp.trueIfNotNull
           case LessThan(_, _) =>
             Not(EqualTo(fromExp, Literal(max, fromType)))
           case GreaterThanOrEqual(_, _) | EqualTo(_, _) | EqualNullSafe(_, _) =>
             EqualTo(fromExp, Literal(max, fromType))
           case _ => exp
         }
   ```
   
   which I think should be pretty fast? comparing to the rule itself which may require walking the whole plan tree.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-683410433






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] sunchao commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479723976



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastSuite.scala
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans.DslLogicalPlan
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types.DoubleType
+
+class UnwrapCastSuite extends PlanTest {
+
+  object Optimize extends RuleExecutor[LogicalPlan] {
+    val batches: List[Batch] =
+      Batch("Unwrap casts", FixedPoint(10), ConstantFolding, UnwrapCast) :: Nil
+  }
+
+  val testRelation: LocalRelation = LocalRelation('a.short, 'b.float)
+
+  test("unwrap casts when literal == max") {
+    val v = Short.MaxValue
+    assertEquivalent('a > v.toInt, UnwrapCast.falseIfNotNull('a))
+    assertEquivalent('a >= v.toInt, 'a === v)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, UnwrapCast.trueIfNotNull('a))
+    assertEquivalent('a < v.toInt, 'a =!= v)
+  }
+
+  test("unwrap casts when literal > max") {
+    val v: Int = Short.MaxValue + 100

Review comment:
       Didn't know this exist. Will switch.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [WIP][SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r478853136



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]

Review comment:
       Is `toType` always `Int`? Can it be `Long` or something else?

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCast.scala
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with the following pattern:
+ *   `BinaryComparison(Cast(fromExp, _), Literal(value, toType))`
+ * This rule optimize expressions with this pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, so they can be
+ * optimized away and pushed down to data sources.
+ *
+ * Currently this only handles the case where `fromType` (of `fromExp`) and `toType` are of
+ * integral types (i.e., byte, short, int and long). It checks to see if the literal `value` is
+ * within range (min, max) of the `fromType`. If this is true then it means we can safely cast the
+ * `value` to the `fromType` and thus able to move the cast to the literal side. Otherwise, it
+ * replaces the cast with different simpler constructs, such as
+ * `EqualTo(fromExp, Literal(max, fromType)` when input is
+ * `GreaterThanOrEqualTo(Cast(fromExp, fromType), Literal(max, toType))`
+ */
+object UnwrapCast extends Rule[LogicalPlan] {
+  override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+    case l: LogicalPlan => l transformExpressionsUp {
+      case e @ BinaryComparison(_, _) => unwrapCast(e)
+    }
+  }
+
+  private def unwrapCast(exp: Expression): Expression = exp match {
+    case BinaryComparison(Literal(_, _), Cast(_, _, _)) =>
+      def swap(e: Expression): Expression = e match {
+        case GreaterThan(left, right) => LessThan(right, left)
+        case GreaterThanOrEqual(left, right) => LessThanOrEqual(right, left)
+        case EqualTo(left, right) => EqualTo(right, left)
+        case EqualNullSafe(left, right) => EqualNullSafe(right, left)
+        case LessThanOrEqual(left, right) => GreaterThanOrEqual(right, left)
+        case LessThan(left, right) => GreaterThan(right, left)
+        case other => other
+      }
+      swap(unwrapCast(swap(exp)))
+
+    case BinaryComparison(Cast(fromExp, _, _), Literal(value, toType)) =>
+      val fromType = fromExp.dataType
+      if (!fromType.isInstanceOf[IntegralType] || !toType.isInstanceOf[IntegralType]
+        || !Cast.canUpCast(fromType, toType)) {
+        return exp
+      }
+
+      // Check if the literal value is within the range of the `fromType`, and handle the boundary
+      // cases in the following
+      val toIntegralType = toType.asInstanceOf[IntegralType]
+      val (min, max) = getRange(fromType)
+      val (minInToType, maxInToType) =
+        (Cast(Literal(min), toType).eval(), Cast(Literal(max), toType).eval())
+
+      // Compare upper bounds
+      val maxCmp = toIntegralType.ordering.asInstanceOf[Ordering[Any]].compare(value, maxInToType)
+      if (maxCmp > 0) {
+        exp match {
+          case EqualTo(_, _) | GreaterThan(_, _) | GreaterThanOrEqual(_, _) =>
+            return falseIfNotNull(fromExp)

Review comment:
       In scala, we don't need `return`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29565: [SPARK-24994][SQL] Simplify casts for literal types

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r479703625



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastSuite.scala
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans.DslLogicalPlan
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types.DoubleType
+
+class UnwrapCastSuite extends PlanTest {
+
+  object Optimize extends RuleExecutor[LogicalPlan] {
+    val batches: List[Batch] =
+      Batch("Unwrap casts", FixedPoint(10), ConstantFolding, UnwrapCast) :: Nil
+  }
+
+  val testRelation: LocalRelation = LocalRelation('a.short, 'b.float)
+
+  test("unwrap casts when literal == max") {
+    val v = Short.MaxValue
+    assertEquivalent('a > v.toInt, UnwrapCast.falseIfNotNull('a))
+    assertEquivalent('a >= v.toInt, 'a === v)
+    assertEquivalent('a === v.toInt, 'a === v)
+    assertEquivalent('a <=> v.toInt, 'a === v)
+    assertEquivalent('a <= v.toInt, UnwrapCast.trueIfNotNull('a))
+    assertEquivalent('a < v.toInt, 'a =!= v)
+  }
+
+  test("unwrap casts when literal > max") {
+    val v: Int = Short.MaxValue + 100

Review comment:
       Shall we use the predefined one instead of adding a random one?
   ```scala
   +import org.apache.spark.sql.catalyst.expressions.IntegralLiteralTestUtils._
   ...
   -    val v: Int = Short.MaxValue + 100
   +    val v: Int = positiveInt
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29565:
URL: https://github.com/apache/spark/pull/29565#issuecomment-690816017






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dbtsai commented on a change in pull request #29565: [SPARK-24994][SQL] Add UnwrapCastInBinaryComparison optimizer to simplify literal types

Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29565:
URL: https://github.com/apache/spark/pull/29565#discussion_r480807290



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.types._
+
+/**
+ * Unwrap casts in binary comparison operations with patterns like following:
+ *
+ * `BinaryComparison(Cast(fromExp, toType), Literal(value, toType))`
+ *   or
+ * `BinaryComparison(Literal(value, toType), Cast(fromExp, toType))`
+ *
+ * This rule optimizes expressions with the above pattern by either replacing the cast with simpler
+ * constructs, or moving the cast from the expression side to the literal side, which enables them
+ * to be optimized away later and pushed down to data sources.
+ *
+ * Currently this only handles cases where `fromType` (of `fromExp`) and `toType` are of integral
+ * types (i.e., byte, short, int and long). The rule checks to see if the literal `value` is
+ * within range `(min, max)`, where `min` and `max` are the minimum and maximum value of
+ * `fromType`, respectively. If this is true then it means we can safely cast `value` to `fromType`
+ * and thus able to move the cast to the literal side.
+ *
+ * If the `value` is not within range `(min, max)`, the rule breaks the scenario into different
+ * cases and try to replace each with simpler constructs.
+ *
+ * if `value > max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) === value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) <=> value` ==> false
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> if(isnull(exp), null, true)
+ *
+ * if `value == max`, the cases are of following:
+ *  - `cast(exp, ty) > value` ==> if(isnull(exp), null, false)
+ *  - `cast(exp, ty) >= value` ==> exp == max
+ *  - `cast(exp, ty) === value` ==> exp == max
+ *  - `cast(exp, ty) <=> value` ==> exp == max
+ *  - `cast(exp, ty) <= value` ==> if(isnull(exp), null, true)
+ *  - `cast(exp, ty) < value` ==> exp =!= max
+ *
+ * Similarly for the cases when `value == min` and `value < min`.

Review comment:
       If we don't have `maxCmp == 0` and `minCmp ==0`, it will be optimized to the following. Those can still be pushed down.
   ```
         val lit = Cast(Literal(value), fromType)
         exp match {
           case GreaterThan(_, _) => GreaterThan(fromExp, lit)
           case GreaterThanOrEqual(_, _) => GreaterThanOrEqual(fromExp, lit)
           case EqualTo(_, _) => EqualTo(fromExp, lit)
           case EqualNullSafe(_, _) => EqualNullSafe(fromExp, lit)
           case LessThan(_, _) => LessThan(fromExp, lit)
           case LessThanOrEqual(_, _) => LessThanOrEqual(fromExp, lit)
           case _ => exp
         }
   ```
   
   Maybe we can have separate rules that optimize from `GreaterThan(fromExp, lit)` to `fromExp.falseIfNotNull` when lit is the max of lit.dataType. So more plans can take advantage of it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org