You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2024/11/18 09:31:47 UTC
(spark) branch master updated: [SPARK-45265][SQL][BUILD][FOLLOWUP] Add `-Xss64m` for Maven testing of the `sql/hive` module
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8b2d0328c213 [SPARK-45265][SQL][BUILD][FOLLOWUP] Add `-Xss64m` for Maven testing of the `sql/hive` module
8b2d0328c213 is described below
commit 8b2d0328c21352e24e4e1c2bbe0a4e97da6a8f2d
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Nov 18 17:31:33 2024 +0800
[SPARK-45265][SQL][BUILD][FOLLOWUP] Add `-Xss64m` for Maven testing of the `sql/hive` module
### What changes were proposed in this pull request?
This pr add `-Xss64m` for Maven testing of the `sql/hive` module
### Why are the changes needed?
Use test options consistent with sbt and fix the mavren daily test
- https://github.com/apache/spark/actions/runs/11879474992/job/33101447051
- https://github.com/apache/spark/actions/runs/11879098676/job/33100585352
- https://github.com/apache/spark/actions/runs/11881833077/job/33106633785
```
*** RUN ABORTED ***
An exception or error caused a run to abort: MetaException(message:Exception rolling back to savepoint rollback_5974738954110)
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Exception rolling back to savepoint rollback_5974738954110)
at org.apache.hadoop.hive.ql.metadata.Hive.addPartitions(Hive.java:3592)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.spark.sql.hive.client.Shim_v4_0.createPartitions(HiveShim.scala:1437)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createPartitions$1(HiveClientImpl.scala:663)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:302)
at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:233)
...
Cause: org.apache.hadoop.hive.metastore.api.MetaException: Exception rolling back to savepoint rollback_5974738954110
at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4508)
at org.apache.hadoop.hive.metastore.ObjectStore.addPartitionsInternal(ObjectStore.java:2811)
at org.apache.hadoop.hive.metastore.ObjectStore.addPartitions(ObjectStore.java:2729)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at jdk.proxy25/jdk.proxy25.$Proxy252.addPartitions(Unknown Source)
at org.apache.hadoop.hive.metastore.HMSHandler.add_partitions_core(HMSHandler.java:4304)
...
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- https://github.com/LuciferYang/spark/runs/33114760431
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #48870 from LuciferYang/SPARK-45265-FOLLOWUP.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Kent Yao <ya...@apache.org>
---
project/SparkBuild.scala | 2 ++
sql/hive/pom.xml | 6 +++++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index f95a95a72b8d..48b243618eea 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1190,6 +1190,8 @@ object Hive {
// smaller size of Xss to mock a FAILED_TO_PARSE_TOO_COMPLEX error, so we need to set for
// hive moudle specifically.
(Test / javaOptions) := (Test / javaOptions).value.filterNot(_.contains("Xss")),
+ // SPARK-45265: The value for `-Xss` should be consistent with the configuration value for
+ // `scalatest-maven-plugin` in `sql/hive/pom.xml`
(Test / javaOptions) += "-Xss64m",
// Supporting all SerDes requires us to depend on deprecated APIs, so we turn off the warnings
// only for this subproject.
diff --git a/sql/hive/pom.xml b/sql/hive/pom.xml
index 5a325f5f56bf..dc0f88beeab8 100644
--- a/sql/hive/pom.xml
+++ b/sql/hive/pom.xml
@@ -237,7 +237,11 @@
<artifactId>scalatest-maven-plugin</artifactId>
<configuration>
<!-- Specially disable assertions since some Hive tests fail them -->
- <argLine>-da -Xmx4g -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}</argLine>
+ <!--
+ SPARK-45265: The value for `-Xss` should be consistent with the configuration value
+ for `Test / javaOptions` in `Hive.settings` within `SparkBuild.scala`.
+ -->
+ <argLine>-da -Xmx4g -Xss64m -XX:ReservedCodeCacheSize=${CodeCacheSize} ${extraJavaTestArgs}</argLine>
</configuration>
</plugin>
<plugin>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org