You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/02/08 02:33:45 UTC
[spark] branch branch-3.0 updated:
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 579e158 [SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
579e158 is described below
commit 579e15873c5a8ff3fcaafec5ce99e37884768f51
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Mon Feb 8 11:32:23 2021 +0900
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
### What changes were proposed in this pull request?
This is a follow-up for SPARK-34346 which causes a flakiness due to `core-site.xml` test resource file addition. This PR aims to remove the test resource `core/src/test/resources/core-site.xml` from `core` module.
### Why are the changes needed?
Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub Action and Jenkins.
```
$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds)
[info] FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:210)
```
To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn` module's `yarn/Client`, but it seems an overkill in `core` module.
```
// SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs so that user's
// environments do not interfere with tests. This allows a special env variable during
// tests so that custom conf dirs can be used by unit tests.
val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++
(if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil)
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs.
Closes #31515 from dongjoon-hyun/SPARK-34346-2.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit dcaf62afea8791e49a44c2062fe14bafdcc0e92f)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
core/src/test/resources/core-site.xml | 24 ----------------------
.../scala/org/apache/spark/SparkContextSuite.scala | 3 ++-
2 files changed, 2 insertions(+), 25 deletions(-)
diff --git a/core/src/test/resources/core-site.xml b/core/src/test/resources/core-site.xml
deleted file mode 100644
index 84eddf8..0000000
--- a/core/src/test/resources/core-site.xml
+++ /dev/null
@@ -1,24 +0,0 @@
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one or more
- ~ contributor license agreements. See the NOTICE file distributed with
- ~ this work for additional information regarding copyright ownership.
- ~ The ASF licenses this file to You under the Apache License, Version 2.0
- ~ (the "License"); you may not use this file except in compliance with
- ~ the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
-
-<configuration>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/tmp/hive_zero</value>
- <description>default is /tmp/hadoop-${user.name} and will be overridden</description>
- </property>
-</configuration>
diff --git a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
index e0a5f52..01b1deb 100644
--- a/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkContextSuite.scala
@@ -979,11 +979,12 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkContext with Eventu
val testKey = "hadoop.tmp.dir"
val bufferKey = "io.file.buffer.size"
val hadoopConf0 = new Configuration()
+ hadoopConf0.set(testKey, "/tmp/hive_zero")
val hiveConfFile = Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
assert(hiveConfFile != null)
hadoopConf0.addResource(hiveConfFile)
- assert(hadoopConf0.get(testKey) === "/tmp/hive_one")
+ assert(hadoopConf0.get(testKey) === "/tmp/hive_zero")
assert(hadoopConf0.get(bufferKey) === "201811")
val sparkConf = new SparkConf()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org