You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Ngone51 <gi...@git.apache.org> on 2018/01/09 08:20:55 UTC
[GitHub] spark pull request #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests ...
GitHub user Ngone51 opened a pull request:
https://github.com/apache/spark/pull/20199
[Spark-22967][Hive]Fix VersionSuite's unit tests by change Windows path into URI path
## What changes were proposed in this pull request?
Two unit test will fail due to Windows format path:
1.test(s"$version: read avro file containing decimal")
```
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);
```
2.test(s"$version: SPARK-17920: Insert into/overwrite avro table")
```
Unable to infer the schema. The schema specification is required to create the table `default`.`tab2`.;
org.apache.spark.sql.AnalysisException: Unable to infer the schema. The schema specification is required to create the table `default`.`tab2`.;
```
This pr fix these two unit test by change Windows path into URI path.
## How was this patch tested?
Existed.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Ngone51/spark SPARK-22967
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20199
----
commit 3d1cafa1c9387b017a98f2983a4e98842a4a5921
Author: wuyi5 <ng...@...>
Date: 2018-01-09T08:01:03Z
change Windows path into URI format path
commit 22669d1ff0cb00261fa146d276af237c115a0488
Author: wuyi5 <ng...@...>
Date: 2018-01-09T08:08:30Z
leave deletion work to ShutdownHookManager
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160408416
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -58,7 +58,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
*/
protected def withTempDir(f: File => Unit): Unit = {
val dir = Utils.createTempDir().getCanonicalFile
- try f(dir) finally Utils.deleteRecursively(dir)
+ f(dir)
--- End diff --
Yea, sounds good.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
Will take a look soon.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20199
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests ...
Posted by Ngone51 <gi...@git.apache.org>.
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160404821
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -58,7 +58,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
*/
protected def withTempDir(f: File => Unit): Unit = {
val dir = Utils.createTempDir().getCanonicalFile
- try f(dir) finally Utils.deleteRecursively(dir)
+ f(dir)
--- End diff --
Actually, only one(2nd test mentioned above). Probably, `assume(!(Utils.isWindows && version == "0.12"))` would be ok. WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
I met the similar case when I fixed some tests related with Kafka before - https://issues-test.apache.org/jira/browse/KAFKA-1194. This took me a while to debug and find this issue ..
To me, I am fine if we fail to find the issue that causes the failure in Hive but might be worth to give a try to find the issue causing this failure and leave the link here.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
Build started: [SQL] `org.apache.spark.sql.hive.client.VersionsSuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/A43BBFC9-F966-40FD-BA66-98D175F432E2)
Diff: https://github.com/apache/spark/compare/master...spark-test:A43BBFC9-F966-40FD-BA66-98D175F432E2
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by Ngone51 <gi...@git.apache.org>.
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20199
Ok, will try.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
Merged to master and branch-2.3.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160396117
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -58,7 +58,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
*/
protected def withTempDir(f: File => Unit): Unit = {
val dir = Utils.createTempDir().getCanonicalFile
- try f(dir) finally Utils.deleteRecursively(dir)
+ f(dir)
--- End diff --
Hm, actually, shall we just skip the failing tests for now (`assume(Utils.isWindows)`)? Given what we talked, only few tests are failed in some few conditions when Hive version is 0.12?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20199
**[Test build #85894 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85894/testReport)** for PR 20199 at commit [`04b6628`](https://github.com/apache/spark/commit/04b6628e310e55acc7b6449f2b48b4752d07a391).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
Let's fix the PR title to `[SPARK-22967][TESTS]`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160681813
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -842,6 +842,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
}
test(s"$version: SPARK-17920: Insert into/overwrite avro table") {
+ assume(!(Utils.isWindows && version == "0.12"))
--- End diff --
@Ngone51 Have you had a chance to check out the issue? If you can't find, let's leave a comment here saying like ... it's intendedly skipped because it fails on Windows.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160879315
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -842,6 +842,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
}
test(s"$version: SPARK-17920: Insert into/overwrite avro table") {
+ assume(!(Utils.isWindows && version == "0.12"))
--- End diff --
Nice. Yea, let's merge this one after adding a comment saying it's skipped because it's failed in the condition on Windows.
Then, see if we can fix the root cause, and then re-enable this test in the PR if everything goes well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20199
**[Test build #85960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85960/testReport)** for PR 20199 at commit [`c3d93c3`](https://github.com/apache/spark/commit/c3d93c3d5c14f080ed5dbf668c0b1241bd82f2fc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85960/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20199
**[Test build #85894 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85894/testReport)** for PR 20199 at commit [`04b6628`](https://github.com/apache/spark/commit/04b6628e310e55acc7b6449f2b48b4752d07a391).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by Ngone51 <gi...@git.apache.org>.
Github user Ngone51 commented on the issue:
https://github.com/apache/spark/pull/20199
cc @HyukjinKwon
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
I can trigger a test on Windows. Let me leave the build:
Build started: [SQL] `org.apache.spark.sql.hive.client.VersionsSuite` [](https://ci.appveyor.com/project/spark-test/spark/branch/68230874-FD64-41B8-82FD-527386DEDE69)
Diff: https://github.com/apache/spark/compare/master...spark-test:68230874-FD64-41B8-82FD-527386DEDE69
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85847/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20199
**[Test build #85847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85847/testReport)** for PR 20199 at commit [`22669d1`](https://github.com/apache/spark/commit/22669d1ff0cb00261fa146d276af237c115a0488).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests ...
Posted by Ngone51 <gi...@git.apache.org>.
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160344054
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -58,7 +58,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
*/
protected def withTempDir(f: File => Unit): Unit = {
val dir = Utils.createTempDir().getCanonicalFile
- try f(dir) finally Utils.deleteRecursively(dir)
+ f(dir)
--- End diff --
Leave deletion work to ShutdownHookManager to avoid delete IOException caused by 'file occupation in other program' error on Windows. (SEE SPARK-22967)
And temp dirs will be cleaned up after unit test completed, but this is only guaranteed for test(s"$version: SPARK-17920: Insert into/overwrite avro table"). And a lot of temp dirs produced by some other unit tests will still remains on Windows for unclear reason, maybe 'file occupation in other program ' too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20199
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85894/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/20199
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests by cha...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20199
**[Test build #85960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85960/testReport)** for PR 20199 at commit [`c3d93c3`](https://github.com/apache/spark/commit/c3d93c3d5c14f080ed5dbf668c0b1241bd82f2fc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20199: [Spark-22967][Hive]Fix VersionSuite's unit tests by chan...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20199
**[Test build #85847 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85847/testReport)** for PR 20199 at commit [`22669d1`](https://github.com/apache/spark/commit/22669d1ff0cb00261fa146d276af237c115a0488).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests...
Posted by Ngone51 <gi...@git.apache.org>.
Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20199#discussion_r160709123
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala ---
@@ -842,6 +842,7 @@ class VersionsSuite extends SparkFunSuite with Logging {
}
test(s"$version: SPARK-17920: Insert into/overwrite avro table") {
+ assume(!(Utils.isWindows && version == "0.12"))
--- End diff --
@HyukjinKwon Ok, will comment.
And I have find serval issues which exactly describe the same problem:
- https://issues.apache.org/jira/browse/SPARK-12216
- https://issues.apache.org/jira/browse/SPARK-8333
- https://issues.apache.org/jira/browse/SPARK-18979
It seems this is a common issue exists on Windows and haven't been resolved yet. And I didn't find a accurate cause for the error from those issues. And now, I doubt this problem maybe related to URLClassLoader. And, I have reproduce the same issue after I did a experiment with URLClassLoader. Though, I'm not sure this is the real cause, yet. Working on this.
We can merge this pr to master after I add comment. And open a new pr of that issue(if fixed) or communicate under one of those issues. WDYT?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org