You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2018/02/12 04:29:55 UTC

[GitHub] spark pull request #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDat...

GitHub user cloud-fan opened a pull request:

    https://github.com/apache/spark/pull/20584

    [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSourceSuite in Spark 2.3/hadoop 2.7

    ## What changes were proposed in this pull request?
    
    This test only fails with sbt on Hadoop 2.7, I can't reproduce it locally, but here is my speculation by looking at the code:
    1. FileSystem.delete doesn't delete the directory entirely, somehow we can still open the file as a 0-length empty file.(just speculation)
    2. ORC intentionally allow empty files, and the reader fails during reading without closing the file stream.
    
    This PR improves the test to make sure all files are deleted and can't be opened.
    
    ## How was this patch tested?
    
    N/A

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cloud-fan/spark flaky-test

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20584.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20584
    
----
commit 51bb48a4189aeb0322dd4ccd0f02416a52e963c3
Author: Wenchen Fan <we...@...>
Date:   2018-02-12T04:24:35Z

    make sure all files are deleted when testing IGNORE_MISSING_FILES

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by gatorsmile <gi...@git.apache.org>.

Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    LGTM, I would merge this first and see whether this can help fix the flaky tests.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    I won't get in the way but I am less sure on this. I thought this is also flaky in PR builder too anyway.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/802/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by HyukjinKwon <gi...@git.apache.org>.

Github user HyukjinKwon commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    You are right. I have run out of ideas. LGTM too for a try if it happens more frequently in spark-branch-2.3-test-sbt-hadoop-2.7.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    **[Test build #87321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87321/testReport)** for PR 20584 at commit [`51bb48a`](https://github.com/apache/spark/commit/51bb48a4189aeb0322dd4ccd0f02416a52e963c3).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by dongjoon-hyun <gi...@git.apache.org>.

Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    My bad. Thank you, guys. For the following, I'll investigate it.
    > According to the log, the leaked file stream was created when building the ORC columnar reader.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    I am also thinking about this. I agree with this.
    > According to the log, the leaked file stream was created when building the ORC columnar reader.
    
    I am suspicious about relationship between `afterEach()` and `addTaskCompletionListener` (call `close()`). But, not sure. Let us try this approach first.



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...

Posted by sameeragarwal <gi...@git.apache.org>.

Github user sameeragarwal commented on the issue:

    https://github.com/apache/spark/pull/20584
  
    merging this to master/2.3. Thanks!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org