You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/01/11 19:51:35 UTC
[spark] branch branch-3.2 updated: [SPARK-37866][TESTS] Set `file.encoding` to UTF-8 for SBT tests
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new deb6776 [SPARK-37866][TESTS] Set `file.encoding` to UTF-8 for SBT tests
deb6776 is described below
commit deb6776b705423141abe92715c3d93b23f082503
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Jan 11 11:48:18 2022 -0800
[SPARK-37866][TESTS] Set `file.encoding` to UTF-8 for SBT tests
This PR aims to set `-Dfile.encoding=UTF-8` for SBT tests.
To make the tests robust on various OSs with different locales.
**BEFORE**
```
$ LANG=C.UTF-8 build/sbt "sql/testOnly *.JDBCV2Suite -- -z non-ascii"
...
[info] - column name with non-ascii *** FAILED *** (2 seconds, 668 milliseconds)
[info] "== Parsed Logical Plan ==
[info] 'Project [unresolvedalias('COUNT('?), None)]
[info] +- 'UnresolvedRelation [h2, test, person], [], false
[info]
[info] == Analyzed Logical Plan ==
[info] count(?): bigint
[info] Aggregate [count(?#x) AS count(?)#xL]
[info] +- SubqueryAlias h2.test.person
[info] +- RelationV2[?#x] test.person
[info]
[info] == Optimized Logical Plan ==
[info] Project [COUNT(U&"\540d")#xL AS count(?#x)#xL AS count(?)#xL]
[info] +- RelationV2[COUNT(U&"\540d")#xL] test.person
[info]
[info] == Physical Plan ==
[info] *(1) Project [COUNT(U&"\540d")#xL AS count(?)#xL]
[info] +- *(1) Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1df881a4 [COUNT(U&"\540d")#xL] PushedAggregates: [COUNT(`?`)], PushedFilters: [], PushedGroupByColumns: [], ReadSchema: struct<COUNT(U&"\540d"):bigint>
[info]
[info] " did not contain "PushedAggregates: [COUNT(`名`)]" (ExplainSuite.scala:66)
```
**AFTER**
```
[info] JDBCV2Suite:
...
[info] - column name with non-ascii (2 seconds, 950 milliseconds)
...
[info] All tests passed.
[success] Total time: 21 s, completed Jan 11, 2022, 2:18:29 AM
```
No.
Pass the CIs.
Closes #35165 from dongjoon-hyun/SPARK-37866.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 122a59986f6ab358452a293256ad45be0fe96a1c)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
project/SparkBuild.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index b1531a6..14af38f 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1127,7 +1127,7 @@ object TestSettings {
// SPARK-29282 This is for consistency between JDK8 and JDK11.
(Test / javaOptions) ++= {
val metaspaceSize = sys.env.get("METASPACE_SIZE").getOrElse("1300m")
- s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads -XX:ReservedCodeCacheSize=128m"
+ s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:+UseParallelGC -XX:-UseDynamicNumberOfGCThreads -XX:ReservedCodeCacheSize=128m -Dfile.encoding=UTF-8"
.split(" ").toSeq
},
javaOptions ++= {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org