You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2021/10/10 23:09:51 UTC
[spark] branch master updated: [SPARK-36900][TESTS][BUILD] Increase
test memory to 6g for Java 11
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6ed1314 [SPARK-36900][TESTS][BUILD] Increase test memory to 6g for Java 11
6ed1314 is described below
commit 6ed13147c99b2f652748b716c70dd1937230cafd
Author: Sean Owen <sr...@gmail.com>
AuthorDate: Sun Oct 10 18:08:37 2021 -0500
[SPARK-36900][TESTS][BUILD] Increase test memory to 6g for Java 11
### What changes were proposed in this pull request?
Increase JVM test memory from 4g to 6g.
### Why are the changes needed?
Running tests under Java 11 fails consistently on a few tests without more memory. The tests do legitimately grab a lot of memory, I believe. It's not super clear why memory usage is different in Java 11, but, also seems fine to just give comfortably more heap to tests for now.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests, run manually with Java 11.
Closes #34214 from srowen/SPARK-36900.
Authored-by: Sean Owen <sr...@gmail.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
pom.xml | 6 +++---
project/SparkBuild.scala | 4 ++--
resource-managers/kubernetes/integration-tests/pom.xml | 2 +-
sql/catalyst/pom.xml | 2 +-
sql/core/pom.xml | 2 +-
sql/hive/pom.xml | 2 +-
6 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/pom.xml b/pom.xml
index 5c6d3a8..1ef33e1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2657,7 +2657,7 @@
<jvmArgs>
<jvmArg>-Xss128m</jvmArg>
<jvmArg>-Xms4g</jvmArg>
- <jvmArg>-Xmx4g</jvmArg>
+ <jvmArg>-Xmx6g</jvmArg>
<jvmArg>-XX:MaxMetaspaceSize=2g</jvmArg>
<jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>
</jvmArgs>
@@ -2707,7 +2707,7 @@
<include>**/*Suite.java</include>
</includes>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx6g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
<environmentVariables>
<!--
Setting SPARK_DIST_CLASSPATH is a simple way to make sure any child processes
@@ -2758,7 +2758,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx6g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
<stderr/>
<environmentVariables>
<!--
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index b1531a6..54d7181 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1127,12 +1127,12 @@ object TestSettings {
// SPARK-29282 This is for consistency between JDK8 and JDK11.
(Test / javaOptions) ++= {
val metaspaceSize = sys.env.get("METASPACE_SIZE").getOrElse("1300m")
- s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads -XX:ReservedCodeCacheSize=128m"
+ s"-Xmx6g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads -XX:ReservedCodeCacheSize=128m"
.split(" ").toSeq
},
javaOptions ++= {
val metaspaceSize = sys.env.get("METASPACE_SIZE").getOrElse("1300m")
- s"-Xmx4g -XX:MaxMetaspaceSize=$metaspaceSize".split(" ").toSeq
+ s"-Xmx6g -XX:MaxMetaspaceSize=$metaspaceSize".split(" ").toSeq
},
(Test / javaOptions) ++= {
val jdwpEnabled = sys.props.getOrElse("test.jdwp.enabled", "false").toBoolean
diff --git a/resource-managers/kubernetes/integration-tests/pom.xml b/resource-managers/kubernetes/integration-tests/pom.xml
index d281e38..20e6e01 100644
--- a/resource-managers/kubernetes/integration-tests/pom.xml
+++ b/resource-managers/kubernetes/integration-tests/pom.xml
@@ -138,7 +138,7 @@
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
- <argLine>-ea -Xmx4g -XX:ReservedCodeCacheSize=1g ${extraScalaTestArgs}</argLine>
+ <argLine>-ea -Xmx6g -XX:ReservedCodeCacheSize=1g ${extraScalaTestArgs}</argLine>
<stderr/>
<systemProperties>
<log4j.configuration>file:src/test/resources/log4j.properties</log4j.configuration>
diff --git a/sql/catalyst/pom.xml b/sql/catalyst/pom.xml
index 32281b5..e457d5a 100644
--- a/sql/catalyst/pom.xml
+++ b/sql/catalyst/pom.xml
@@ -158,7 +158,7 @@
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<configuration>
- <argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx6g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
</configuration>
</plugin>
<plugin>
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index d2a31b7..44a5539 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -233,7 +233,7 @@
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<configuration>
- <argLine>-ea -Xmx4g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-ea -Xmx6g -Xss4m -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
</configuration>
</plugin>
<plugin>
diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml
index 86cd1ed..4146753 100644
--- a/sql/hive/pom.xml
+++ b/sql/hive/pom.xml
@@ -234,7 +234,7 @@
<artifactId>scalatest-maven-plugin</artifactId>
<configuration>
<!-- Specially disable assertions since some Hive tests fail them -->
- <argLine>-da -Xmx4g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
+ <argLine>-da -Xmx6g -XX:ReservedCodeCacheSize=${CodeCacheSize} -Dio.netty.tryReflectionSetAccessible=true</argLine>
</configuration>
</plugin>
<plugin>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org