You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2018/02/01 18:26:30 UTC

[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

GitHub user dongjoon-hyun opened a pull request:

    https://github.com/apache/spark/pull/20479

    [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMissingFiles` test case for ORC

    ## What changes were proposed in this pull request?
    
    Like Parquet, Apache Spark ORC already handles `spark.sql.files.ignoreMissingFiles` correctly. We had better have a test coverage for feature parity.
    
    ## How was this patch tested?
    
    Pass Jenkins with a newly added test case.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dongjoon-hyun/spark SPARK-23305

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20479.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20479
    
----
commit 7fdfc35e5fc0f92b4bf2f687606bb48ad6fa68e6
Author: Dongjoon Hyun <do...@...>
Date:   2018-02-01T18:23:00Z

    [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMissingFiles` test case for ORC

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86955/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86940 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86940/testReport)** for PR 20479 at commit [`7fdfc35`](https://github.com/apache/spark/commit/7fdfc35e5fc0f92b4bf2f687606bb48ad6fa68e6).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #87019 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87019/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #87022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87022/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86976/testReport)** for PR 20479 at commit [`4512d72`](https://github.com/apache/spark/commit/4512d72bfdea574cf7b7b5fae412dba3afb6f88a).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86982 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86982/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86976/testReport)** for PR 20479 at commit [`4512d72`](https://github.com/apache/spark/commit/4512d72bfdea574cf7b7b5fae412dba3afb6f88a).
     * This patch **fails due to an unknown error code, -9**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165576607
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
    @@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  // Only ORC/Parquet support this.
    +  Seq("orc", "parquet").foreach { format =>
    --- End diff --
    
    Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #87022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87022/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87022/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Ah, I moved both of them into `FileBasedDataSourceSuite`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Thank you for review and approval!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/558/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86982 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86982/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86940 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86940/testReport)** for PR 20479 at commit [`7fdfc35`](https://github.com/apache/spark/commit/7fdfc35e5fc0f92b4bf2f687606bb48ad6fa68e6).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Retest this please.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    @gatorsmile . Can we do the refactoring later since we have still many things to do for RC3?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86940/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165557435
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
    @@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  testQuietly("Enabling/disabling ignoreMissingFiles") {
    --- End diff --
    
    Basically, this is copied from Parquet. To avoid duplicate codes, create a common base test class for parquet and orc? Then, we can deduplicate the codes? 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165537214
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
    @@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  testQuietly("Enabling/disabling ignoreMissingFiles") {
    +    def testIgnoreMissingFiles(): Unit = {
    +      withTempDir { dir =>
    +        val basePath = dir.getCanonicalPath
    +        spark.range(1).toDF("a").write.orc(new Path(basePath, "first").toString)
    +        spark.range(1, 2).toDF("a").write.orc(new Path(basePath, "second").toString)
    +        val thirdPath = new Path(basePath, "third")
    +        spark.range(2, 3).toDF("a").write.orc(thirdPath.toString)
    +        val df = spark.read.orc(
    +          new Path(basePath, "first").toString,
    +          new Path(basePath, "second").toString,
    +          new Path(basePath, "third").toString)
    +
    +        val fs = thirdPath.getFileSystem(spark.sparkContext.hadoopConfiguration)
    +        fs.delete(thirdPath, true)
    --- End diff --
    
    Sure.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86955/testReport)** for PR 20479 at commit [`c2891d0`](https://github.com/apache/spark/commit/c2891d05df4c6f158eccac3ba96a934bcb0dfbb4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Thank you, @HyukjinKwon and @gatorsmile .


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/497/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #87019 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87019/testReport)** for PR 20479 at commit [`9a2f640`](https://github.com/apache/spark/commit/9a2f640f3e695970c0c4ffe93d6fe978c4013ed1).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Thanks! Merged to master/2.3


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165577600
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
    @@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  // Only ORC/Parquet support this.
    --- End diff --
    
    Wait .. don't other sources support this option?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/556/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165579611
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
    @@ -92,4 +96,37 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  allFileBasedDataSources.foreach { format =>
    --- End diff --
    
    Thank you so much, @HyukjinKwon !


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    I updated the JIRA issue and PR descriptions to mention all file-based data sources.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Thank you for review, @HyukjinKwon , @viirya , and @gatorsmile .
    Now, there is no redundancy.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86982/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165575853
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
    @@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  // Only ORC/Parquet support this.
    +  Seq("orc", "parquet").foreach { format =>
    --- End diff --
    
    Ah, yeah. This sounds better.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165578302
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala ---
    @@ -92,4 +96,39 @@ class FileBasedDataSourceSuite extends QueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  // Only ORC/Parquet support this.
    --- End diff --
    
    Ur, let me check back.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    **[Test build #86955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86955/testReport)** for PR 20479 at commit [`c2891d0`](https://github.com/apache/spark/commit/c2891d05df4c6f158eccac3ba96a934bcb0dfbb4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/516/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86976/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165528709
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
    @@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  testQuietly("Enabling/disabling ignoreMissingFiles") {
    +    def testIgnoreMissingFiles(): Unit = {
    +      withTempDir { dir =>
    +        val basePath = dir.getCanonicalPath
    +        spark.range(1).toDF("a").write.orc(new Path(basePath, "first").toString)
    +        spark.range(1, 2).toDF("a").write.orc(new Path(basePath, "second").toString)
    +        val thirdPath = new Path(basePath, "third")
    +        spark.range(2, 3).toDF("a").write.orc(thirdPath.toString)
    +        val df = spark.read.orc(
    +          new Path(basePath, "first").toString,
    +          new Path(basePath, "second").toString,
    +          new Path(basePath, "third").toString)
    +
    +        val fs = thirdPath.getFileSystem(spark.sparkContext.hadoopConfiguration)
    +        fs.delete(thirdPath, true)
    --- End diff --
    
    Shall we assert `true`?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/521/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ig...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/20479


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ign...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20479#discussion_r165559636
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala ---
    @@ -655,4 +655,35 @@ class OrcQuerySuite extends OrcQueryTest with SharedSQLContext {
           }
         }
       }
    +
    +  testQuietly("Enabling/disabling ignoreMissingFiles") {
    --- End diff --
    
    +1. Which suite is proper for that base test class?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Test `spark.sql.files.ignoreMis...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87019/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/485/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #20479: [SPARK-23305][SQL][TEST] Add `spark.sql.files.ignoreMiss...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20479
  
    For test-only PRs, we still can merge it to 2.3


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org