[spark] branch branch-3.0 updated: [SPARK-33292][SQL] Make Literal ArrayBasedMapData string representation disambiguous

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new 563a678  [SPARK-33292][SQL] Make Literal ArrayBasedMapData string representation disambiguous
563a678 is described below

commit 563a678bc276fd6ead1f7858e0e37373133f4ac2
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Thu Oct 29 19:10:01 2020 -0700

    [SPARK-33292][SQL] Make Literal ArrayBasedMapData string representation disambiguous
    
    ### What changes were proposed in this pull request?
    
    This PR aims to wrap `ArrayBasedMapData` literal representation with `map(...)`.
    
    ### Why are the changes needed?
    
    Literal ArrayBasedMapData has inconsistent string representation from `LogicalPlan` to `Optimized Logical Plan/Physical Plan`. Also, the representation at `Optimized Logical Plan` and `Physical Plan` is ambiguous like `[1 AS a#0, keys: [key1], values: [value1] AS b#1]`.
    
    **BEFORE**
    ```scala
    scala> spark.version
    res0: String = 2.4.7
    
    scala> sql("SELECT 1 a, map('key1', 'value1') b").explain(true)
    == Parsed Logical Plan ==
    'Project [1 AS a#0, 'map(key1, value1) AS b#1]
    +- OneRowRelation
    
    == Analyzed Logical Plan ==
    a: int, b: map<string,string>
    Project [1 AS a#0, map(key1, value1) AS b#1]
    +- OneRowRelation
    
    == Optimized Logical Plan ==
    Project [1 AS a#0, keys: [key1], values: [value1] AS b#1]
    +- OneRowRelation
    
    == Physical Plan ==
    *(1) Project [1 AS a#0, keys: [key1], values: [value1] AS b#1]
    +- Scan OneRowRelation[]
    ```
    
    **AFTER**
    ```scala
    scala> spark.version
    res0: String = 3.1.0-SNAPSHOT
    
    scala> sql("SELECT 1 a, map('key1', 'value1') b").explain(true)
    == Parsed Logical Plan ==
    'Project [1 AS a#4, 'map(key1, value1) AS b#5]
    +- OneRowRelation
    
    == Analyzed Logical Plan ==
    a: int, b: map<string,string>
    Project [1 AS a#4, map(key1, value1) AS b#5]
    +- OneRowRelation
    
    == Optimized Logical Plan ==
    Project [1 AS a#4, map(keys: [key1], values: [value1]) AS b#5]
    +- OneRowRelation
    
    == Physical Plan ==
    *(1) Project [1 AS a#4, map(keys: [key1], values: [value1]) AS b#5]
    +- *(1) Scan OneRowRelation[]
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. This changes the query plan's string representation in `explain` command and UI. However, this is a bug fix.
    
    ### How was this patch tested?
    
    Pass the CI with the newly added test case.
    
    Closes #30190 from dongjoon-hyun/SPARK-33292.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
    (cherry picked from commit 838791bf0b8290143001fe8f94b1fbbd53a181d2)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala  | 1 +
 .../apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala   | 1 +
 2 files changed, 2 insertions(+)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
index 213a58a..9e96ab8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
@@ -297,6 +297,7 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
   override def toString: String = value match {
     case null => "null"
     case binary: Array[Byte] => s"0x" + DatatypeConverter.printHexBinary(binary)
+    case d: ArrayBasedMapData => s"map(${d.toString})"
     case other => other.toString
   }
 
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
index 4714635..bb86135 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
@@ -209,6 +209,7 @@ class LiteralExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
     }
     checkMapLiteral(Map("a" -> 1, "b" -> 2, "c" -> 3))
     checkMapLiteral(Map("1" -> 1.0, "2" -> 2.0, "3" -> 3.0))
+    assert(Literal.create(Map("a" -> 1)).toString === "map(keys: [a], values: [1])")
   }
 
   test("struct") {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org