[spark] branch master updated: [SPARK-36373][SQL] DecimalPrecision only add necessary cast

[yu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yumwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new c20af53  [SPARK-36373][SQL] DecimalPrecision only add necessary cast
c20af53 is described below

commit c20af535803a7250fef047c2bf0fe30be242369d
Author: Yuming Wang <yu...@ebay.com>
AuthorDate: Tue Aug 3 08:12:54 2021 +0800

    [SPARK-36373][SQL] DecimalPrecision only add necessary cast
    
    ### What changes were proposed in this pull request?
    
    This pr makes `DecimalPrecision` only add necessary cast similar to [`ImplicitTypeCasts`](https://github.com/apache/spark/blob/96c2919988ddf78d104103876d8d8221e8145baa/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala#L675-L678). For example:
    ```
    EqualTo(AttributeReference("d1", DecimalType(5, 2))(), AttributeReference("d2", DecimalType(2, 1))())
    ```
    It will add a useless cast to _d1_:
    ```
    (cast(d1#6 as decimal(5,2)) = cast(d2#7 as decimal(5,2)))
    ```
    
    ### Why are the changes needed?
    
    1. Avoid adding unnecessary cast. Although it will be removed by  `SimplifyCasts` later.
    2. I'm trying to add an extended rule similar to `PullOutGroupingExpressions`. The current behavior will introduce additional alias. For example: `cast(d1 as decimal(5,2)) as cast_d1`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Unit test.
    
    Closes #33602 from wangyum/SPARK-36373.
    
    Authored-by: Yuming Wang <yu...@ebay.com>
    Signed-off-by: Yuming Wang <yu...@ebay.com>
---
 .../org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala     | 4 +++-
 .../apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala    | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala
index bf128cd..b112641 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecision.scala
@@ -204,7 +204,9 @@ object DecimalPrecision extends TypeCoercionRule {
     case b @ BinaryComparison(e1 @ DecimalType.Expression(p1, s1),
     e2 @ DecimalType.Expression(p2, s2)) if p1 != p2 || s1 != s2 =>
       val resultType = widerDecimalType(p1, s1, p2, s2)
-      b.makeCopy(Array(Cast(e1, resultType), Cast(e2, resultType)))
+      val newE1 = if (e1.dataType == resultType) e1 else Cast(e1, resultType)
+      val newE2 = if (e2.dataType == resultType) e2 else Cast(e2, resultType)
+      b.makeCopy(Array(newE1, newE2))
   }
 
   /**
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala
index 834f38a..698d2d5 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionSuite.scala
@@ -59,6 +59,9 @@ class DecimalPrecisionSuite extends AnalysisTest with BeforeAndAfter {
     val comparison = analyzer.execute(plan).collect {
       case Project(Alias(e: BinaryComparison, _) :: Nil, _) => e
     }.head
+    // Only add necessary cast.
+    assert(comparison.left.children.forall(_.dataType !== expectedType))
+    assert(comparison.right.children.forall(_.dataType !== expectedType))
     assert(comparison.left.dataType === expectedType)
     assert(comparison.right.dataType === expectedType)
   }

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org