[GitHub] [spark] sumeetgajjar commented on a change in pull request #35047: [SPARK-37175][SQL] Performance improvement to hash joins with many duplicate keys

[GitBox <gi...@apache.org>]

sumeetgajjar commented on a change in pull request #35047:
URL: https://github.com/apache/spark/pull/35047#discussion_r779786446



##########
File path: project/SparkBuild.scala
##########
@@ -1141,7 +1141,8 @@ object TestSettings {
     (Test / javaOptions) += "-ea",
     (Test / javaOptions) ++= {
       val metaspaceSize = sys.env.get("METASPACE_SIZE").getOrElse("1300m")
-      val extraTestJavaArgs = Array("-XX:+IgnoreUnrecognizedVMOptions",

Review comment:
       The actual change is to expose an ENV variable to set the heap size. If set this heap size will be using as `-Xmx` while invoking `runMain` from sbt. 
   The default 4g headspace was not sufficient when I was running the newly added benchmark for 10 billion rows (driver OOM'd), thus the change.
   
   While making the change I noticed `extraTestJavaArgs` `Array` was first converted to a string and immediately at the next line it was again split on spaces and converted to `Seq`.
   Thus I refactored the code a bit.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org