You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/16 21:33:46 UTC
(spark) branch branch-4.2 updated: [SPARK-60296][MLLIB] Make `DenseMatrix.hashCode` consistent with `equals`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new ef0a3e422e57 [SPARK-60296][MLLIB] Make `DenseMatrix.hashCode` consistent with `equals`
ef0a3e422e57 is described below
commit ef0a3e422e57513cda219101bfa7fbec634306b8
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Fri Oct 16 14:29:44 2026 -0700
[SPARK-60296][MLLIB] Make `DenseMatrix.hashCode` consistent with `equals`
### What changes were proposed in this pull request?
This PR aims to make `DenseMatrix.hashCode` value-based and consistent with `equals` in both `org.apache.spark.ml.linalg` and `org.apache.spark.mllib.linalg`.
- `ml.linalg.DenseMatrix.hashCode` now computes the same hash code as `SparseMatrix.hashCode`, i.e., Breeze's `QuasiTensor.hashCode`. It starts from `43` and mixes the `##` of each value with `MurmurHash3.mix` in column-major order, skipping zero hash codes such as those of `0.0` and `-0.0`. A transposed matrix is also visited in column-major order.
- `mllib.linalg.DenseMatrix.hashCode` delegates to `asML.hashCode`.
### Why are the changes needed?
`DenseMatrix.hashCode` hashes `toArray`, which allocates a new array on every call, and an array is hashed by its identity. As a result, the same instance returns a different hash code on every call, and equal matrices have different hash codes. This violates the `hashCode` contract, so a `DenseMatrix` cannot be found in hash-based collections such as `HashSet` and `HashMap`. This has been the case since Apache Spark [v1.4.0](https://github.com/apache/spark/releases/tag/v1.4.0) (2015- [...]
For example, RDD operations silently return wrong results when equal matrices are used as keys. `org.apache.spark.mllib.linalg.DenseMatrix` has the same problem.
```scala
val m = new org.apache.spark.ml.linalg.DenseMatrix(2, 2, Array(1.0, 2.0, 3.0, 4.0))
val rdd = sc.parallelize(Seq(m, m.copy, m.copy, m.toSparse), 2)
rdd.distinct().count() // Before: 4, After: 1
rdd.countByValue().values.sum // Before: 1, After: 4
val left = sc.parallelize(Seq(m -> "left"), 2)
val right = sc.parallelize(Seq(m.copy -> "right"), 2)
left.join(right).count() // Before: 0, After: 1
```
Note that `asBreeze.hashCode()` cannot be used for `DenseMatrix` because Breeze's `DenseMatrix` visits its values in row-major order while `CSCMatrix` visits them in column-major order.
### Does this PR introduce _any_ user-facing change?
Yes. `DenseMatrix.hashCode` becomes deterministic and consistent with `equals`, including the equality between a dense matrix and a sparse matrix with the same entries.
```scala
val m = new org.apache.spark.ml.linalg.DenseMatrix(2, 2, Array(1.0, 2.0, 3.0, 4.0))
m.hashCode == m.hashCode // Before: false, After: true
m == m.toSparse // Before: true, After: true
m.hashCode == m.toSparse.hashCode // Before: false, After: true
scala.collection.mutable.HashSet(m).contains(m) // Before: false, After: true
```
### How was this patch tested?
Pass the CIs with the newly added test cases in `ml.linalg.MatricesSuite` and `mllib.linalg.MatricesSuite`. Without this fix, both new test cases fail at the first assertion, `dm.hashCode === dm.hashCode`.
```
$ build/sbt 'mllib-local/testOnly *MatricesSuite'
$ build/sbt 'mllib/testOnly org.apache.spark.mllib.linalg.MatricesSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59559 from dongjoon-hyun/SPARK-60296.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 979bb1094a2fab9cfa6616bca8286b7e44b3a7ee)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../org/apache/spark/ml/linalg/Matrices.scala | 16 ++++++++++-
.../org/apache/spark/ml/linalg/MatricesSuite.scala | 31 ++++++++++++++++++++++
.../org/apache/spark/mllib/linalg/Matrices.scala | 6 ++---
.../apache/spark/mllib/linalg/MatricesSuite.scala | 23 ++++++++++++++++
4 files changed, 71 insertions(+), 5 deletions(-)
diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
index a783d1af437d..55e109162542 100644
--- a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
+++ b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
@@ -21,6 +21,7 @@ import java.util.{Arrays, Random}
import scala.collection.immutable
import scala.collection.mutable.{ArrayBuffer, ArrayBuilder => MArrayBuilder, HashSet => MHashSet}
+import scala.util.hashing.MurmurHash3
import breeze.linalg.{CSCMatrix => BSM, DenseMatrix => BDM, Matrix => BM}
@@ -332,7 +333,20 @@ class DenseMatrix @Since("2.0.0") (
}
override def hashCode: Int = {
- Seq(numRows, numCols, toArray).##
+ // Matches SparseMatrix.hashCode (Breeze's QuasiTensor.hashCode) so that equal dense and
+ // sparse matrices have the same hash code. The values are visited in column major order
+ // and those with a zero hash code, such as 0.0 and -0.0, are skipped.
+ val data = if (isTransposed) toArray else values
+ var hash = 43
+ var i = 0
+ while (i < data.length) {
+ val valueHash = data(i).##
+ if (valueHash != 0) {
+ hash = MurmurHash3.mix(hash, valueHash)
+ }
+ i += 1
+ }
+ hash
}
private[ml] def asBreeze: BM[Double] = {
diff --git a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala b/mllib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala
index 9003c89334ce..9de9cbf5b130 100644
--- a/mllib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala
+++ b/mllib-local/src/test/scala/org/apache/spark/ml/linalg/MatricesSuite.scala
@@ -123,6 +123,37 @@ class MatricesSuite extends SparkMLFunSuite {
assert(!sparseMat.toArray.eq(sparseCopy.toArray))
}
+ test("SPARK-60296: DenseMatrix.hashCode is consistent with equals") {
+ val dm = new DenseMatrix(3, 2, Array(1.0, -0.0, 2.0, 0.0, 3.0, 0.0))
+ assert(dm.hashCode === dm.hashCode)
+ assert(dm.copy === dm)
+ assert(dm.copy.hashCode === dm.hashCode)
+
+ // The same matrix with the values stored in row major.
+ val dmt = new DenseMatrix(3, 2, Array(1.0, 0.0, -0.0, 3.0, 2.0, 0.0), isTransposed = true)
+ val dmtColMajor = dmt.toDenseColMajor
+ assert(!dmtColMajor.isTransposed)
+ assert(dmt === dmtColMajor)
+ assert(dmt.hashCode === dmtColMajor.hashCode)
+
+ // The same matrix with an explicit 0.0 and an explicit -0.0.
+ val sm = new SparseMatrix(3, 2, Array(0, 3, 5), Array(0, 1, 2, 0, 1),
+ Array(1.0, 0.0, 2.0, -0.0, 3.0))
+ Seq(dm, dmt).foreach { m =>
+ assert(m === sm)
+ assert(m.hashCode === sm.hashCode)
+ assert(m.transpose === sm.transpose)
+ assert(m.transpose.hashCode === sm.transpose.hashCode)
+ }
+
+ // A nonzero value whose hash code is 0 is skipped as well.
+ val x = 4294967297.0 // (1L << 32) + 1
+ assert(x.## === 0)
+ val dmx = new DenseMatrix(2, 1, Array(1.0, x))
+ assert(dmx === dmx.toSparse)
+ assert(dmx.hashCode === dmx.toSparse.hashCode)
+ }
+
test("matrix indexing and updating") {
val m = 3
val n = 2
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
index eff849b533ff..7cd851de91a5 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
@@ -17,7 +17,7 @@
package org.apache.spark.mllib.linalg
-import java.util.{Arrays, Objects, Random}
+import java.util.{Arrays, Random}
import scala.collection.mutable.{ArrayBuffer, ArrayBuilder => MArrayBuilder, HashSet => MHashSet}
import scala.language.implicitConversions
@@ -313,9 +313,7 @@ class DenseMatrix @Since("1.3.0") (
case _ => false
}
- override def hashCode: Int = {
- Objects.hash(numRows: Integer, numCols: Integer, toArray)
- }
+ override def hashCode: Int = asML.hashCode
private[mllib] def asBreeze: BM[Double] = {
if (!isTransposed) {
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala b/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
index d6de9a826900..d5f77c8f3ed6 100644
--- a/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
@@ -151,6 +151,29 @@ class MatricesSuite extends SparkFunSuite {
assert(!sparseMat.toArray.eq(sparseCopy.toArray))
}
+ test("SPARK-60296: DenseMatrix.hashCode is consistent with equals") {
+ val dm = new DenseMatrix(3, 2, Array(1.0, -0.0, 2.0, 0.0, 3.0, 0.0))
+ assert(dm.hashCode === dm.hashCode)
+ assert(dm.copy === dm)
+ assert(dm.copy.hashCode === dm.hashCode)
+
+ // The same matrix with the values stored in row major.
+ val dmt = new DenseMatrix(3, 2, Array(1.0, 0.0, -0.0, 3.0, 2.0, 0.0), isTransposed = true)
+ val dmtColMajor = new DenseMatrix(3, 2, dmt.toArray)
+ assert(dmt === dmtColMajor)
+ assert(dmt.hashCode === dmtColMajor.hashCode)
+
+ // The same matrix with an explicit 0.0 and an explicit -0.0.
+ val sm = new SparseMatrix(3, 2, Array(0, 3, 5), Array(0, 1, 2, 0, 1),
+ Array(1.0, 0.0, 2.0, -0.0, 3.0))
+ Seq(dm, dmt).foreach { m =>
+ assert(m === sm)
+ assert(m.hashCode === sm.hashCode)
+ assert(m.transpose === sm.transpose)
+ assert(m.transpose.hashCode === sm.transpose.hashCode)
+ }
+ }
+
test("matrix indexing and updating") {
val m = 3
val n = 2
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org