You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "George314159 (via GitHub)" <gi...@apache.org> on 2024/09/16 22:16:53 UTC
[PR] [SPARK-49491] Replace AnyRefMap with HashMap [spark]
George314159 opened a new pull request, #48128:
URL: https://github.com/apache/spark/pull/48128
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Replace AnyRefMap with HashMap
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
HashMap has better performance in Scala 2.13
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Existing Unit Tests
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2357196173
Oh, it seems that I mislead you.
- When I wrote `attach`, I meant to add a commit overwriting the below file.
- https://github.com/apache/spark/blob/master/sql/core/benchmarks/DateTimeRebaseBenchmark-results.txt
- In addition, you need to run Java 21 too to update this file.
- https://github.com/apache/spark/blob/master/sql/core/benchmarks/DateTimeRebaseBenchmark-jdk21-results.txt
In Apache Spark community, your contribution (the code and benchmark results) will be merged as a single commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "George314159 (via GitHub)" <gi...@apache.org>.
George314159 commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1762245898
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -22,7 +22,7 @@ import java.time.temporal.ChronoField
import java.util.{Calendar, TimeZone}
import java.util.Calendar.{DAY_OF_MONTH, DST_OFFSET, ERA, HOUR_OF_DAY, MINUTE, MONTH, SECOND, YEAR, ZONE_OFFSET}
-import scala.collection.mutable.AnyRefMap
+import scala.collection.mutable
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2357455717
Thank you for updating the benchmark result.
BTW, according to the existing benchmark result, there is no noticeable difference. It's a little different from 40% difference of https://github.com/apache/spark/pull/48128#issuecomment-2356347119 . Do you have any other insights on the GitHub Action result, @wangyum ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491] Replace AnyRefMap with HashMap [spark]
Posted by "George314159 (via GitHub)" <gi...@apache.org>.
George314159 commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354130400
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2356446552
Since this is `RebaseDateTime.scala`, cc @MaxGekk , too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "George314159 (via GitHub)" <gi...@apache.org>.
George314159 commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2357232700
Got it, thanks for the guidance! I have updated the benchmark results in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "George314159 (via GitHub)" <gi...@apache.org>.
George314159 commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2357145892
[DateTimeRebaseBenchmark-results.txt](https://github.com/user-attachments/files/17036261/DateTimeRebaseBenchmark-results.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1899929263
##########
sql/core/benchmarks/DateTimeRebaseBenchmark-jdk21-results.txt:
##########
@@ -6,109 +6,109 @@ OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 11193 11193 0 8.9 111.9 1.0X
-before 1582, noop 7070 7070 0 14.1 70.7 1.6X
-after 1582, rebase EXCEPTION 19836 19836 0 5.0 198.4 0.6X
-after 1582, rebase LEGACY 19368 19368 0 5.2 193.7 0.6X
-after 1582, rebase CORRECTED 19627 19627 0 5.1 196.3 0.6X
-before 1582, rebase LEGACY 16301 16301 0 6.1 163.0 0.7X
-before 1582, rebase CORRECTED 15612 15612 0 6.4 156.1 0.7X
+after 1582, noop 11734 11734 0 8.5 117.3 1.0X
+before 1582, noop 7155 7155 0 14.0 71.6 1.6X
+after 1582, rebase EXCEPTION 20374 20374 0 4.9 203.7 0.6X
+after 1582, rebase LEGACY 19681 19681 0 5.1 196.8 0.6X
+after 1582, rebase CORRECTED 20160 20160 0 5.0 201.6 0.6X
+before 1582, rebase LEGACY 16245 16245 0 6.2 162.5 0.7X
+before 1582, rebase CORRECTED 16039 16039 0 6.2 160.4 0.7X
Review Comment:
https://github.com/apache/spark/blob/6099de7f5995b438335a5a545d5cf62dd3d68f0d/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DateTimeRebaseBenchmark.scala#L166
@MaxGekk
It seems that the numIters for some bench cases is fixed at 1, so the Stdev is always 0 because there is only test result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2566789264
Thank you all. Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2401385845
> Is this one ready to go?
@LuciferYang See @dongjoon-hyun's comment:
"
We can merge your benchmark suite before @George314159 's PR. Then, compare the result before and after.
"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1762503937
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -22,7 +22,7 @@ import java.time.temporal.ChronoField
import java.util.{Calendar, TimeZone}
import java.util.Calendar.{DAY_OF_MONTH, DST_OFFSET, ERA, HOUR_OF_DAY, MINUTE, MONTH, SECOND, YEAR, ZONE_OFFSET}
-import scala.collection.mutable.AnyRefMap
+import scala.collection.mutable
Review Comment:
Aside from the original goals of the PR itself, personally, I tend to write it as follows:
```
import scala.collection.mutable
val hashMap = new mutable.HashMap[String, RebaseInfo]
```
rather than
```
import scala.collection.mutable.HashMap
val hashMap = new HashMap[String, RebaseInfo]
```
This way, when we need to import `immutable.HashMap` in this class, it won't cause a naming conflict (generally, for immutable data structures, there's no need to add an extra prefix). Additionally, using `mutable.HashMap` makes the code clearer when reading it.
Of course, if due to other historical reasons we must write it in the second way, then that is also feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2552711505
@George314159 Could you rebase the code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354756826
> In addition, Apple Silicon benchmark result is quite misleading because it's not the same Server-side result. That's one of the reasons why Apache Spark community decided to run on GitHub Action. If we have a benchmark in Spark codebase, why don't we regenerate the result in this PR, @George314159 ?
+1 for run the benchmark on GitHub Action.
Locally, I obtained the opposite result after switching the execution order of `HashMap` and `LongMap`.
```scala
...
benchmark.addCase("LongMap") { _ =>
keys.foreach { key => map2.getOrElseUpdate(key, new Object()) }
}
benchmark.addCase("HashMap") { _ =>
keys.foreach { key => map1.getOrElseUpdate(key, new Object()) }
}
...
```
```
OpenJDK 64-Bit Server VM 17.0.12+7-LTS on Mac OS X 14.6.1
Apple M3
Benchmark Map: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
LongMap 443 450 8 0.0 110752.9 1.0X
HashMap 546 785 101 0.0 136431.3 0.8X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1766340208
##########
sql/core/benchmarks/DateTimeRebaseBenchmark-jdk21-results.txt:
##########
@@ -6,109 +6,109 @@ OpenJDK 64-Bit Server VM 21.0.4+7-LTS on Linux 6.5.0-1025-azure
AMD EPYC 7763 64-Core Processor
Save DATE to parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-after 1582, noop 11193 11193 0 8.9 111.9 1.0X
-before 1582, noop 7070 7070 0 14.1 70.7 1.6X
-after 1582, rebase EXCEPTION 19836 19836 0 5.0 198.4 0.6X
-after 1582, rebase LEGACY 19368 19368 0 5.2 193.7 0.6X
-after 1582, rebase CORRECTED 19627 19627 0 5.1 196.3 0.6X
-before 1582, rebase LEGACY 16301 16301 0 6.1 163.0 0.7X
-before 1582, rebase CORRECTED 15612 15612 0 6.4 156.1 0.7X
+after 1582, noop 11734 11734 0 8.5 117.3 1.0X
+before 1582, noop 7155 7155 0 14.0 71.6 1.6X
+after 1582, rebase EXCEPTION 20374 20374 0 4.9 203.7 0.6X
+after 1582, rebase LEGACY 19681 19681 0 5.1 196.8 0.6X
+after 1582, rebase CORRECTED 20160 20160 0 5.0 201.6 0.6X
+before 1582, rebase LEGACY 16245 16245 0 6.2 162.5 0.7X
+before 1582, rebase CORRECTED 16039 16039 0 6.2 160.4 0.7X
Review Comment:
Not related to this PR, but `Stdev` is all `0` confuses slightly.
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -285,12 +285,12 @@ object RebaseDateTime {
}
// Loads rebasing info from an JSON file. JSON records in the files should conform to
- // `JsonRebaseRecord`. AnyRefMap is used here instead of Scala's immutable map because
- // it is 2 times faster in DateTimeRebaseBenchmark.
- private[sql] def loadRebaseRecords(fileName: String): AnyRefMap[String, RebaseInfo] = {
+ // `JsonRebaseRecord`. Mutable HashMap is used here instead of AnyRefMap due to SPARK-49491.
+ private[sql] def loadRebaseRecords(fileName: String): HashMap[String, RebaseInfo] = {
val file = SparkClassUtils.getSparkClassLoader.getResource(fileName)
val jsonRebaseRecords = mapper.readValue[Seq[JsonRebaseRecord]](file)
- val anyRefMap = new AnyRefMap[String, RebaseInfo]((3 * jsonRebaseRecords.size) / 2)
+ val hashMap = new HashMap[String, RebaseInfo]
+ hashMap.sizeHint((3 * jsonRebaseRecords.size) / 2)
Review Comment:
Please, set just `jsonRebaseRecords.size`. `HashMap` has some internal heuristic already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2358681901
Then, could you contribute your code as a new benchmark suite to Apache Spark repository, @wangyum ? I believe it's worthy to have. 😄 We can merge your benchmark suite before @George314159 's PR. Then, compare the result before and after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1762503937
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -22,7 +22,7 @@ import java.time.temporal.ChronoField
import java.util.{Calendar, TimeZone}
import java.util.Calendar.{DAY_OF_MONTH, DST_OFFSET, ERA, HOUR_OF_DAY, MINUTE, MONTH, SECOND, YEAR, ZONE_OFFSET}
-import scala.collection.mutable.AnyRefMap
+import scala.collection.mutable
Review Comment:
Aside from the original goals of the PR itself, personally, I tend to write it as follows:
```
import scala.collection.mutable
val hashMap = new mutable.HashMap[String, RebaseInfo]
```
rather than
```
import scala.collection.mutable.HashMap
val hashMap = new HashMap[String, RebaseInfo]
```
This way, when we need to import `immutable.HashMap` in this class, it won't cause a naming conflict (generally, for immutable data structures, there's no need to add an extra prefix). Additionally, using `mutable.HashMap` makes the code clearer when reading it."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2401414222
Set this one to draft first to avoid it being accidentally merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2566780927
@LuciferYang You need specific tests to show performance differences, such as: https://github.com/apache/spark/pull/48128#issuecomment-2356347119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum closed pull request #48128: [SPARK-49491][SQL] Replace AnyRefMap with HashMap
URL: https://github.com/apache/spark/pull/48128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2356347119
> To @wangyum , do you have the benchmark result with `2.13.14` because Apache Spark 4.0.0 is using Scala 2.13.14.
scala 2.13.14:
```
OpenJDK 64-Bit Server VM 17.0.7+7-LTS on Mac OS X 14.5
Apple M2 Max
Benchmark Map: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
HashMap 854 862 6 0.0 213583.8 1.0X
AnyRefMap 1444 1455 7 0.0 361086.9 0.6X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1762224657
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -285,12 +285,12 @@ object RebaseDateTime {
}
// Loads rebasing info from an JSON file. JSON records in the files should conform to
- // `JsonRebaseRecord`. AnyRefMap is used here instead of Scala's immutable map because
- // it is 2 times faster in DateTimeRebaseBenchmark.
- private[sql] def loadRebaseRecords(fileName: String): AnyRefMap[String, RebaseInfo] = {
+ // `JsonRebaseRecord`. Mutable HashMap is used here instead of AnyRefMap due to SPARK-49491.
+ private[sql] def loadRebaseRecords(fileName: String): mutable.HashMap[String, RebaseInfo] = {
Review Comment:
```suggestion
// `JsonRebaseRecord`. HashMap is used here instead of Scala's immutable map because
// it is 2 times faster in DateTimeRebaseBenchmark.
private[sql] def loadRebaseRecords(fileName: String): HashMap[String, RebaseInfo] = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354430832
+1. `HashMap` is much faster than `AnyRefMap` since Scala 2.13.0
scala 2.13.8:
```
OpenJDK 64-Bit Server VM 1.8.0_382-b05 on Mac OS X 14.5
Apple M2 Max
Benchmark Map: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
HashMap 1121 1126 3 0.0 280264.2 1.0X
AnyRefMap 1590 1609 29 0.0 397386.4 0.7X
```
scala 2.12.18:
```
OpenJDK 64-Bit Server VM 1.8.0_382-b05 on Mac OS X 14.5
Apple M2 Max
Benchmark Map: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
HashMap 1679 1689 11 0.0 419759.2 1.0X
AnyRefMap 1585 1592 8 0.0 396357.1 1.1X
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on code in PR #48128:
URL: https://github.com/apache/spark/pull/48128#discussion_r1762223675
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:
##########
@@ -22,7 +22,7 @@ import java.time.temporal.ChronoField
import java.util.{Calendar, TimeZone}
import java.util.Calendar.{DAY_OF_MONTH, DST_OFFSET, ERA, HOUR_OF_DAY, MINUTE, MONTH, SECOND, YEAR, ZONE_OFFSET}
-import scala.collection.mutable.AnyRefMap
+import scala.collection.mutable
Review Comment:
```suggestion
import scala.collection.mutable.HashMap
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354483754
In addition, Apple Silicon benchmark result is quite misleading because it's not the same Server-side result. That's one of the reasons why Apache Spark community decided to run on GitHub Action. If we have a benchmark in Spark codebase, why don't we regenerate the result in this PR, @George314159 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2566154767
@wangyum Which benchmark's results should we update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354193160
cc @LuciferYang and @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2356451804
To @George314159 , please read `Running benchmarks in your forked repository` section of the community guideline.
- https://spark.apache.org/developer-tools.html
Then, please attach the regenerate result of the following Benchmark.
- https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DateTimeRebaseBenchmark.scala
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354770718
@wangyum @George314159
```scala
val benchmark = new Benchmark("Benchmark Map", size, minNumIters = 30)
benchmark.addCase("HashMap") { _ =>
keys.foreach { key => map1.getOrElseUpdate(key, new Object()) }
}
```
Is this benchmark primarily testing the performance of the `get`? After the first iteration, all key-value pairs are already in the map, so subsequent iterations should not reach the `OrElseUpdate` branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2354481876
To @wangyum , do you have the benchmark result with `2.13.14` because Apache Spark 4.0.0 is using Scala 2.13.14.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2401412675
> > Is this one ready to go?
>
> @LuciferYang See @dongjoon-hyun's comment: " We can merge your benchmark suite before @George314159 's PR. Then, compare the result before and after. "
Thanks for your clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49491][SQL] Replace AnyRefMap with HashMap [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #48128:
URL: https://github.com/apache/spark/pull/48128#issuecomment-2357563382
@dongjoon-hyun It seems the `HashMap` is very small in `DateTimeRebaseBenchmark`. For more details about `HashMap` vs `AnyRefMap`, please see https://github.com/scala/bug/issues/13029.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org