You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2018/02/01 18:26:30 UTC
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/20479
[SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMissingFiles` test case for ORC
## What changes were proposed in this pull request?
Like Parquet, Apache Spark ORC already handles `spark.sql.files.ignoreMissingFiles` correctly. We had better have a test coverage for feature parity.
## How was this patch tested?
Pass Jenkins with a newly added test case.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-23305
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20479.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20479
----
commit 7fdfc35e5fc0f92b4bf2f687606bb48ad6fa68e6
Author: Dongjoon Hyun <do...@...>
Date: 2018-02-01T18:23:00Z
[SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMissingFiles` test case for ORC
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86955/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86940 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86940/testReport)** for PR 20479 at commit [`7fdfc35`](https://github.com/apache/spark/commit/7fdfc35e5fc0f92b4bf2f687606bb48ad6fa68e6).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #87019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87019/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #87022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87022/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86976/testReport)** for PR 20479 at commit [`4512d72`](https://github.com/apache/spark/commit/4512d72bfdea574cf7b7b5fae412dba3afb6f88a).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86982 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86982/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86976/testReport)** for PR 20479 at commit [`4512d72`](https://github.com/apache/spark/commit/4512d72bfdea574cf7b7b5fae412dba3afb6f88a).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165576607
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
@@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
}
}
}
+
+ // Only ORC/Parquet support this.
+ Seq("orc", "parquet").foreach { format =>
--- End diff --
Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #87022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87022/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87022/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
Ah, I moved both of them into `FileBasedDataSourceSuite`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
Thank you for review and approval!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/558/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86982/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86940/testReport)** for PR 20479 at commit [`7fdfc35`](https://github.com/apache/spark/commit/7fdfc35e5fc0f92b4bf2f687606bb48ad6fa68e6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
@gatorsmile . Can we do the refactoring later since we have still many things to do for RC3?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86940/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165557435
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
@@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
}
}
}
+
+ testQuietly("Enabling/disabling ignoreMissingFiles") {
--- End diff --
Basically, this is copied from Parquet. To avoid duplicate codes, create a common base test class for parquet and orc? Then, we can deduplicate the codes?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165537214
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
@@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
}
}
}
+
+ testQuietly("Enabling/disabling ignoreMissingFiles") {
+ def testIgnoreMissingFiles(): Unit = {
+ withTempDir { dir =>
+ val basePath = dir.getCanonicalPath
+ spark.range(1).toDF("a").write.orc(new Path(basePath, "first").toString)
+ spark.range(1, 2).toDF("a").write.orc(new Path(basePath, "second").toString)
+ val thirdPath = new Path(basePath, "third")
+ spark.range(2, 3).toDF("a").write.orc(thirdPath.toString)
+ val df = spark.read.orc(
+ new Path(basePath, "first").toString,
+ new Path(basePath, "second").toString,
+ new Path(basePath, "third").toString)
+
+ val fs = thirdPath.getFileSystem(spark.sparkContext.hadoopConfiguration)
+ fs.delete(thirdPath, true)
--- End diff --
Sure.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86955/testReport)** for PR 20479 at commit [`c2891d0`](https://github.com/apache/spark/commit/c2891d05df4c6f158eccac3ba96a934bcb0dfbb4).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
Thank you, @HyukjinKwon and @gatorsmile .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/497/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #87019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87019/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20479
Thanks! Merged to master/2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165577600
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
@@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
}
}
}
+
+ // Only ORC/Parquet support this.
--- End diff --
Wait .. don't other sources support this option?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/556/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165579611
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
@@ -92,4 +96,37 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
}
}
}
+
+ allFileBasedDataSources.foreach { format =>
--- End diff --
Thank you so much, @HyukjinKwon !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
I updated the JIRA issue and PR descriptions to mention all file-based data sources.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20479
Thank you for review, @HyukjinKwon , @viirya , and @gatorsmile .
Now, there is no redundancy.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86982/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165575853
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
@@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
}
}
}
+
+ // Only ORC/Parquet support this.
+ Seq("orc", "parquet").foreach { format =>
--- End diff --
Ah, yeah. This sounds better.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165578302
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
@@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
}
}
}
+
+ // Only ORC/Parquet support this.
--- End diff --
Ur, let me check back.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/20479
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20479
**[Test build #86955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86955/testReport)** for PR 20479 at commit [`c2891d0`](https://github.com/apache/spark/commit/c2891d05df4c6f158eccac3ba96a934bcb0dfbb4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20479
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/516/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86976/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165528709
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
@@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
}
}
}
+
+ testQuietly("Enabling/disabling ignoreMissingFiles") {
+ def testIgnoreMissingFiles(): Unit = {
+ withTempDir { dir =>
+ val basePath = dir.getCanonicalPath
+ spark.range(1).toDF("a").write.orc(new Path(basePath, "first").toString)
+ spark.range(1, 2).toDF("a").write.orc(new Path(basePath, "second").toString)
+ val thirdPath = new Path(basePath, "third")
+ spark.range(2, 3).toDF("a").write.orc(thirdPath.toString)
+ val df = spark.read.orc(
+ new Path(basePath, "first").toString,
+ new Path(basePath, "second").toString,
+ new Path(basePath, "third").toString)
+
+ val fs = thirdPath.getFileSystem(spark.sparkContext.hadoopConfiguration)
+ fs.delete(thirdPath, true)
--- End diff --
Shall we assert `true`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/521/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ig...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20479
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20479#discussion_r165559636
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
@@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
}
}
}
+
+ testQuietly("Enabling/disabling ignoreMissingFiles") {
--- End diff --
+1. Which suite is proper for that base test class?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87019/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20479
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/485/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20479
For test-only PRs, we still can merge it to 2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org