You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by viirya <gi...@git.apache.org> on 2018/01/30 06:06:03 UTC

[GitHub] spark pull request #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not f...

GitHub user viirya opened a pull request:

    https://github.com/apache/spark/pull/20431

    [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

    ## What changes were proposed in this pull request?
    
    It is reported that the test `Cancelling stage in a query with Range` in `DataFrameRangeSuite` fails a few times in unrelated PRs. I personally also saw it too in my PR.
    
    This test is not very flaky actually but only fails occasionally. Based on how the test works, I guess that is because `range` finishes before the listener calls `cancelStage`.
    
    I increase the range number from `1000000000L` to `100000000000L` and count the range in one partition. I also reduce the `interval` of checking stage id. Hopefully it can make the test not flaky anymore.
    
    ## How was this patch tested?
    
    The modified tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/viirya/spark-1 SPARK-23222

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/20431.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #20431
    
----
commit 9a4a4842b3f8281e73e564f4dfdad92017630760
Author: Liang-Chi Hsieh <vi...@...>
Date:   2018-01-30T05:49:01Z

    Make DataFrameRangeSuite not flaky.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

Posted by cloud-fan <gi...@git.apache.org>.

Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/20431
  
    thanks, merging to master/2.3!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

Posted by viirya <gi...@git.apache.org>.

Github user viirya commented on the issue:

    https://github.com/apache/spark/pull/20431
  
    I didn't notice significant difference. It is about 450~500 milliseconds.
    



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #20431: [SPARK-23222][SQL] Make DataFrameRangeSuite not f...

Posted by viirya <gi...@git.apache.org>.

Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20431#discussion_r164670510
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala ---
    @@ -154,7 +154,7 @@ class DataFrameRangeSuite extends QueryTest with SharedSQLContext with Eventuall
       test("Cancelling stage in a query with Range.") {
         val listener = new SparkListener {
           override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
    -        eventually(timeout(10.seconds)) {
    +        eventually(timeout(10.seconds), interval(1.millis)) {
    --- End diff --
    
    The default interval is 15millis, IIUC. It is more possibly that the range stage finishes in the interval. So reduce it to 1millis.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org