You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/08/10 03:53:39 UTC
(spark) branch branch-4.x updated: [SPARK-58682][DOC] Update outdated `OpenHashMap`/`OpenHashSet` performance claims
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 640ada8560e5 [SPARK-58682][DOC] Update outdated `OpenHashMap`/`OpenHashSet` performance claims
640ada8560e5 is described below
commit 640ada8560e510229a64e7347e00ff663adbb2dc
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Sun Aug 9 20:53:08 2026 -0700
[SPARK-58682][DOC] Update outdated `OpenHashMap`/`OpenHashSet` performance claims
### What changes were proposed in this pull request?
This PR aims to update the outdated performance claims in the class documentation of
`OpenHashMap` and `OpenHashSet`, based on the results of `OpenHashMapBenchmark` added
by SPARK-58678.
- `OpenHashMap`: replaced the claim "about 5X faster than java.util.HashMap" with an
accurate description: it uses much less space and is competitive for aggregation
workloads (`changeValue`), while `java.util.HashMap` is faster for pure insertions
and lookups on modern JDKs.
- `OpenHashSet`: replaced the unqualified claim "much faster than Java's standard
HashSet" with the verifiable facts: much less memory overhead, and the specialized
versions avoid boxing of primitive keys.
### Why are the changes needed?
The "about 5X faster" claim dates from 2013 (pre-JDK 8). `OpenHashMapBenchmark`
(SPARK-58678) shows that on modern JDKs the claim no longer holds:
- **Insert**: `java.util.HashMap` is 4.6X, 5.5X, 6.5X faster on Java 17, 21, and 25,
respectively.
- **Lookup**: `java.util.HashMap` is about 2X faster.
- **Aggregate** (`changeValue`/`merge`): `OpenHashMap` is slightly faster.
Since the benchmark's insert/lookup paths with object keys effectively measure
`OpenHashSet` as well, its unqualified "much faster" claim is also updated. The
memory advantage (about 2.2x less for `String -> Long` entries) remains true and is
kept.
### Does this PR introduce _any_ user-facing change?
No. This is a comment-only change.
### How was this patch tested?
Pass the CIs. This is a comment-only change with no behavior change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57885 from dongjoon-hyun/SPARK-58682.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 5adcd21e6073236b57ab2871d52df0a0e08c41d5)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../main/scala/org/apache/spark/util/collection/OpenHashMap.scala | 5 +++--
.../main/scala/org/apache/spark/util/collection/OpenHashSet.scala | 7 ++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala b/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
index e421a1f4746e..3c7f083381c3 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala
@@ -21,8 +21,9 @@ import scala.reflect.ClassTag
/**
* A fast hash map implementation for nullable keys. This hash map supports insertions and updates,
- * but not deletions. This map is about 5X faster than java.util.HashMap, while using much less
- * space overhead.
+ * but not deletions. This map uses much less space than java.util.HashMap and is competitive with
+ * it for aggregation workloads (`changeValue`), while java.util.HashMap is faster for pure
+ * insertions and lookups on modern JDKs. See `OpenHashMapBenchmark` for details.
*
* Under the hood, it uses our OpenHashSet implementation.
*
diff --git a/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala b/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
index 3d1eb5788c70..c56ae0224740 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala
@@ -28,9 +28,10 @@ import org.apache.spark.annotation.Private
* removed.
*
* The underlying implementation uses Scala compiler's specialization to generate optimized
- * storage for four primitive types (Long, Int, Double, and Float). It is much faster than Java's
- * standard HashSet while incurring much less memory overhead. This can serve as building blocks
- * for higher level data structures such as an optimized HashMap.
+ * storage for four primitive types (Long, Int, Double, and Float). It incurs much less memory
+ * overhead than Java's standard HashSet, and the specialized versions avoid boxing of primitive
+ * keys. This can serve as building blocks for higher level data structures such as an optimized
+ * HashMap.
*
* This OpenHashSet is designed to serve as building blocks for higher level data structures
* such as an optimized hash map. Compared with standard hash set implementations, this class
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org