You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/01/11 19:50:16 UTC
[spark] branch master updated: [SPARK-37866][TESTS] Set `file.encoding` to UTF-8 for SBT tests
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 122a599 [SPARK-37866][TESTS] Set `file.encoding` to UTF-8 for SBT tests
122a599 is described below
commit 122a59986f6ab358452a293256ad45be0fe96a1c
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Jan 11 11:48:18 2022 -0800
[SPARK-37866][TESTS] Set `file.encoding` to UTF-8 for SBT tests
### What changes were proposed in this pull request?
This PR aims to set `-Dfile.encoding=UTF-8` for SBT tests.
### Why are the changes needed?
To make the tests robust on various OSs with different locales.
**BEFORE**
```
$ LANG=C.UTF-8 build/sbt "sql/testOnly *.JDBCV2Suite -- -z non-ascii"
...
[info] - column name with non-ascii *** FAILED *** (2 seconds, 668 milliseconds)
[info] "== Parsed Logical Plan ==
[info] 'Project [unresolvedalias('COUNT('?), None)]
[info] +- 'UnresolvedRelation [h2, test, person], [], false
[info]
[info] == Analyzed Logical Plan ==
[info] count(?): bigint
[info] Aggregate [count(?#x) AS count(?)#xL]
[info] +- SubqueryAlias h2.test.person
[info] +- RelationV2[?#x] test.person
[info]
[info] == Optimized Logical Plan ==
[info] Project [COUNT(U&"\540d")#xL AS count(?#x)#xL AS count(?)#xL]
[info] +- RelationV2[COUNT(U&"\540d")#xL] test.person
[info]
[info] == Physical Plan ==
[info] *(1) Project [COUNT(U&"\540d")#xL AS count(?)#xL]
[info] +- *(1) Scan org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCScan$$anon$1df881a4 [COUNT(U&"\540d")#xL] PushedAggregates: [COUNT(`?`)], PushedFilters: [], PushedGroupByColumns: [], ReadSchema: struct<COUNT(U&"\540d"):bigint>
[info]
[info] " did not contain "PushedAggregates: [COUNT(`名`)]" (ExplainSuite.scala:66)
```
**AFTER**
```
[info] JDBCV2Suite:
...
[info] - column name with non-ascii (2 seconds, 950 milliseconds)
...
[info] All tests passed.
[success] Total time: 21 s, completed Jan 11, 2022, 2:18:29 AM
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #35165 from dongjoon-hyun/SPARK-37866.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
project/SparkBuild.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index b5b713c..4130c6a 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1155,7 +1155,7 @@ object TestSettings {
"--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
"--add-opens=java.base/sun.security.action=ALL-UNNAMED",
"--add-opens=java.base/sun.util.calendar=ALL-UNNAMED").mkString(" ")
- s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:ReservedCodeCacheSize=128m $extraTestJavaArgs"
+ s"-Xmx4g -Xss4m -XX:MaxMetaspaceSize=$metaspaceSize -XX:ReservedCodeCacheSize=128m -Dfile.encoding=UTF-8 $extraTestJavaArgs"
.split(" ").toSeq
},
javaOptions ++= {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org