You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/01/15 09:06:00 UTC
[jira] [Assigned] (SPARK-46721) Some tests failed when cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5
[ https://issues.apache.org/jira/browse/SPARK-46721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot reassigned SPARK-46721:
--------------------------------------
Assignee: (was: Apache Spark)
> Some tests failed when cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-46721
> URL: https://issues.apache.org/jira/browse/SPARK-46721
> Project: Spark
> Issue Type: Test
> Components: Tests
> Affects Versions: 4.0.0
> Reporter: Bobby Wang
> Priority: Trivial
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> When cherry-picking https://github.com/apache/spark/pull/43494 back to branch 3.5 https://github.com/apache/spark/pull/44690,
> I ran into the issue that some tests for Scala 2.12 failed when comparing two maps. It turned out that the function [compareMaps](https://github.com/apache/spark/pull/43494/files#diff-f205431247dd9446f4ce941e5a4620af438c242b9bdff6e7faa7df0194db49acR129) is not so robust for scala 2.12 and scala 2.13.
> - scala 2.13
> ``` scala
> Welcome to Scala 2.13.12 (OpenJDK 64-Bit Server VM, Java 17.0.9).
> Type in expressions for evaluation. Or try :help.
> scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
> | eps: Double = 0.00000001): Boolean = {
> | lhs.size == rhs.size &&
> | lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>
> | lName == rName && (lAmount - rAmount).abs < eps
> | }
> | }
> |
> | import scala.collection.mutable.HashMap
> | val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, "d"-> 4.0))
> | val mapped = resources.map { case (rName, addressAmounts) =>
> | rName -> HashMap(addressAmounts.toSeq.sorted: _*)
> | }
> |
> | compareMaps(resources("gpu"), mapped("gpu").toMap)
> def compareMaps(lhs: Map[String,Double], rhs: Map[String,Double], eps: Double): Boolean
> import scala.collection.mutable.HashMap
> val resources: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
> val mapped: scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] = Map(gpu -> HashMap(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
> val res0: Boolean = true
> ```
> - scala 2.12
> ``` scala
> Welcome to Scala 2.12.14 (OpenJDK 64-Bit Server VM, Java 17.0.9).
> Type in expressions for evaluation. Or try :help.
> scala> def compareMaps(lhs: Map[String, Double], rhs: Map[String, Double],
> | eps: Double = 0.00000001): Boolean = {
> | lhs.size == rhs.size &&
> | lhs.zip(rhs).forall { case ((lName, lAmount), (rName, rAmount)) =>
> | lName == rName && (lAmount - rAmount).abs < eps
> | }
> | }
> compareMaps: (lhs: Map[String,Double], rhs: Map[String,Double], eps: Double)Boolean
> scala> import scala.collection.mutable.HashMap
> import scala.collection.mutable.HashMap
> scala> val resources = Map("gpu" -> Map("a" -> 1.0, "b" -> 2.0, "c" -> 3.0, "d"-> 4.0))
> resources: scala.collection.immutable.Map[String,scala.collection.immutable.Map[String,Double]] = Map(gpu -> Map(a -> 1.0, b -> 2.0, c -> 3.0, d -> 4.0))
> scala> val mapped = resources.map { case (rName, addressAmounts) =>
> | rName -> HashMap(addressAmounts.toSeq.sorted: _*)
> | }
> mapped: scala.collection.immutable.Map[String,scala.collection.mutable.HashMap[String,Double]] = Map(gpu -> Map(b -> 2.0, d -> 4.0, a -> 1.0, c -> 3.0))
> scala> compareMaps(resources("gpu"), mapped("gpu").toMap)
> res0: Boolean = false
> ```
> The same code bug got different results for Scala 2.12 and Scala 2.13. This PR tried to rework compareMaps to make tests pass for both scala 2.12 and scala 2.13
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org