[spark] branch branch-2.4 updated: [SPARK-32377][SQL][2.4] CaseInsensitiveMap should be deterministic for addition

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-2.4 by this push:
     new df930e1  [SPARK-32377][SQL][2.4] CaseInsensitiveMap should be deterministic for addition
df930e1 is described below

commit df930e17fd9c3be02110290b1c68c82c07ee08dd
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Jul 21 19:52:49 2020 +0900

    [SPARK-32377][SQL][2.4] CaseInsensitiveMap should be deterministic for addition
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix `CaseInsensitiveMap` to be deterministic for addition.
    This is a backporting of https://github.com/apache/spark/pull/29172 .
    
    ### Why are the changes needed?
    
    ```scala
    import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
    var m = CaseInsensitiveMap(Map.empty[String, String])
    Seq(("paTh", "1"), ("PATH", "2"), ("Path", "3"), ("patH", "4"), ("path", "5")).foreach { kv =>
      m = (m + kv).asInstanceOf[CaseInsensitiveMap[String]]
      println(m.get("path"))
    }
    ```
    
    **BEFORE**
    ```
    Some(1)
    Some(2)
    Some(3)
    Some(4)
    Some(1)
    ```
    
    **AFTER**
    ```
    Some(1)
    Some(2)
    Some(3)
    Some(4)
    Some(5)
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, but this is a bug fix on non-deterministic behavior.
    
    ### How was this patch tested?
    
    Pass the newly added test case.
    
    Closes #29175 from dongjoon-hyun/SPARK-32377-2.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: HyukjinKwon <gu...@apache.org>
---
 .../sql/catalyst/util/CaseInsensitiveMap.scala     |  2 +-
 .../catalyst/util/CaseInsensitiveMapSuite.scala    | 30 ++++++++++++++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala
index bb2c592..4b149d2 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMap.scala
@@ -36,7 +36,7 @@ class CaseInsensitiveMap[T] private (val originalMap: Map[String, T]) extends Ma
     keyLowerCasedMap.contains(k.toLowerCase(Locale.ROOT))
 
   override def +[B1 >: T](kv: (String, B1)): Map[String, B1] = {
-    new CaseInsensitiveMap(originalMap + kv)
+    new CaseInsensitiveMap(originalMap.filter(!_._1.equalsIgnoreCase(kv._1)) + kv)
   }
 
   override def iterator: Iterator[(String, T)] = keyLowerCasedMap.iterator
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMapSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMapSuite.scala
new file mode 100644
index 0000000..cc6e5b0
--- /dev/null
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/CaseInsensitiveMapSuite.scala
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.SparkFunSuite
+
+class CaseInsensitiveMapSuite extends SparkFunSuite {
+  test("SPARK-32377: CaseInsensitiveMap should be deterministic for addition") {
+     var m = CaseInsensitiveMap(Map.empty[String, String])
+     Seq(("paTh", "1"), ("PATH", "2"), ("Path", "3"), ("patH", "4"), ("path", "5")).foreach { kv =>
+       m = (m + kv).asInstanceOf[CaseInsensitiveMap[String]]
+       assert(m.get("path").contains(kv._2))
+     }
+  }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org