You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by kayousterhout <gi...@git.apache.org> on 2017/02/11 06:16:01 UTC

[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

GitHub user kayousterhout opened a pull request:

    https://github.com/apache/spark/pull/16892

    [SPARK-19560] Improve DAGScheduler tests.

    This commit improves the tests that check the case when a
    ShuffleMapTask completes successfully on an executor that has
    failed.  This commit improves the commenting around the existing
    test for this, and adds some additional checks to make it more
    clear what went wrong if the tests fail (the fact that these
    tests are hard to understand came up in the context of @markhamstra's
    proposed fix for #16620).
    
    This commit also removes a test that I realized tested exactly
    the same functionality.
    
    @markhamstra, I verified that the new version of the test still fails (and
    in a more helpful way) for your proposed change for #16620.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kayousterhout/spark-1 SPARK-19560

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/16892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #16892
    
----
commit 06b26d3738403da30bbf2c7ef2bfdb86039baa02
Author: Kay Ousterhout <ka...@gmail.com>
Date:   2017-02-11T06:04:11Z

    [SPARK-19560] Improve DAGScheduler tests.
    
    This commit improves the tests that check the case when a
    ShuffleMapTask completes successfully on an executor that has
    failed.  This commit improves the commenting around the existing
    test for this, and adds some additional checks to make it more
    clear what went wrong if the tests fail (the fact that these
    tests are hard to understand came up in the context of #16620).
    
    This commit also removes a test that I realized tested exactly
    the same functionality.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16892#discussion_r102821009
  
    --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
    @@ -1569,24 +1569,44 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with Timeou
         assertDataStructuresEmpty()
       }
     
    -  test("run trivial shuffle with out-of-band failure and retry") {
    +  /**
    +   * In this test, we run a map stage where one of the executors fails but we still receive a
    +   * "zombie" complete message from a task that ran on that executor. We want to make sure the
    +   * stage is resubmitted so that the task that ran on the failed executor is re-executed, and
    +   * that the stage is only marked as finished once that task completes.
    +   */
    +  test("run trivial shuffle with out-of-band executor failure and retry") {
         val shuffleMapRdd = new MyRDD(sc, 2, Nil)
         val shuffleDep = new ShuffleDependency(shuffleMapRdd, new HashPartitioner(2))
         val shuffleId = shuffleDep.shuffleId
         val reduceRdd = new MyRDD(sc, 1, List(shuffleDep), tracker = mapOutputTracker)
         submit(reduceRdd, Array(0))
    -    // blockManagerMaster.removeExecutor("exec-hostA")
    -    // pretend we were told hostA went away
    +    // Tell the DAGScheduler that hostA was lost.
         runEvent(ExecutorLost("exec-hostA", ExecutorKilled))
    -    // DAGScheduler will immediately resubmit the stage after it appears to have no pending tasks
    -    // rather than marking it is as failed and waiting.
         complete(taskSets(0), Seq(
           (Success, makeMapStatus("hostA", 1)),
           (Success, makeMapStatus("hostB", 1))))
    +
    +    // At this point, no more tasks are running for the stage (and the TaskSetManager considers the
    +    // stage complete), but the tasks that ran on HostA need to be re-run, so the DAGScheduler
    +    // should re-submit the stage.
    +    assert(taskSets.size === 2)
    +
    +    // Make sure that the stage that was re-submitted was the ShuffleMapStage (not the reduce
    +    // stage, which shouldn't be run until all of the tasks in the ShuffleMapStage complete on
    +    // alive executors).
    +    assert(taskSets(1).tasks(0).isInstanceOf[ShuffleMapTask])
    --- End diff --
    
    Good idea done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Jenkins retest this please (filed https://issues.apache.org/jira/browse/SPARK-19613 for the flaky test)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    cc @mateiz, whose test I deleted / rolled into the existing one


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Ok I added back the other test but improved the commenting there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16892#discussion_r103017616
  
    --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
    @@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with Timeou
        * In this test, we run a map stage where one of the executors fails but we still receive a
        * "zombie" complete message from that executor. We want to make sure the stage is not reported
        * as done until all tasks have completed.
    +   *
    +   * Most of the functionality in this test is tested in "run trivial shuffle with out-of-band
    +   * executor failure and retry".  However, that test uses ShuffleMapStages that are followed by
    +   * a ResultStage, whereas in this test, the ShuffleMapStage is tested in isolation, without a
    +   * ResultStage after it.
    --- End diff --
    
     I see that makes sense.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16892#discussion_r102282650
  
    --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
    @@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with Timeou
        * In this test, we run a map stage where one of the executors fails but we still receive a
        * "zombie" complete message from that executor. We want to make sure the stage is not reported
        * as done until all tasks have completed.
    +   *
    +   * Most of the functionality in this test is tested in "run trivial shuffle with out-of-band
    +   * executor failure and retry".  However, that test uses ShuffleMapStages that are followed by
    +   * a ResultStage, whereas in this test, the ShuffleMapStage is tested in isolation, without a
    +   * ResultStage after it.
    --- End diff --
    
    I should have looked closer at this test earlier ... I was hoping this was testing a *multi*-stage mapjob.  That would really be a better test.  ideally we'd even have three stages, with a failure happening in the second stage, and the last stage.
    
    In any case, your changes still look good, no need to have to do those other things now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72744/testReport)** for PR 16892 at commit [`06b26d3`](https://github.com/apache/spark/commit/06b26d3738403da30bbf2c7ef2bfdb86039baa02).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72732/testReport)** for PR 16892 at commit [`06b26d3`](https://github.com/apache/spark/commit/06b26d3738403da30bbf2c7ef2bfdb86039baa02).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72948/testReport)** for PR 16892 at commit [`f8407f2`](https://github.com/apache/spark/commit/f8407f22b8a763a83b3b8e91a7a0660829d655c1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16892#discussion_r102996593
  
    --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
    @@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with Timeou
        * In this test, we run a map stage where one of the executors fails but we still receive a
        * "zombie" complete message from that executor. We want to make sure the stage is not reported
        * as done until all tasks have completed.
    +   *
    +   * Most of the functionality in this test is tested in "run trivial shuffle with out-of-band
    +   * executor failure and retry".  However, that test uses ShuffleMapStages that are followed by
    +   * a ResultStage, whereas in this test, the ShuffleMapStage is tested in isolation, without a
    +   * ResultStage after it.
    --- End diff --
    
    the *final* map-stage can't have anything that follows it, but the job overall can still have multiple stages, and the failure can occur during the processing of those earlier map-stages, or the final one.
    
    In any case, I agree you don't need to expand that test in this PR.  and even though this test doesn't do as much as I was hoping, I do still think it adds value, and is worth leaving in, even though its very similar to the other test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16892#discussion_r102281985
  
    --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
    @@ -1569,24 +1569,44 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with Timeou
         assertDataStructuresEmpty()
       }
     
    -  test("run trivial shuffle with out-of-band failure and retry") {
    +  /**
    +   * In this test, we run a map stage where one of the executors fails but we still receive a
    +   * "zombie" complete message from a task that ran on that executor. We want to make sure the
    +   * stage is resubmitted so that the task that ran on the failed executor is re-executed, and
    +   * that the stage is only marked as finished once that task completes.
    +   */
    +  test("run trivial shuffle with out-of-band executor failure and retry") {
         val shuffleMapRdd = new MyRDD(sc, 2, Nil)
         val shuffleDep = new ShuffleDependency(shuffleMapRdd, new HashPartitioner(2))
         val shuffleId = shuffleDep.shuffleId
         val reduceRdd = new MyRDD(sc, 1, List(shuffleDep), tracker = mapOutputTracker)
         submit(reduceRdd, Array(0))
    -    // blockManagerMaster.removeExecutor("exec-hostA")
    -    // pretend we were told hostA went away
    +    // Tell the DAGScheduler that hostA was lost.
         runEvent(ExecutorLost("exec-hostA", ExecutorKilled))
    -    // DAGScheduler will immediately resubmit the stage after it appears to have no pending tasks
    -    // rather than marking it is as failed and waiting.
         complete(taskSets(0), Seq(
           (Success, makeMapStatus("hostA", 1)),
           (Success, makeMapStatus("hostB", 1))))
    +
    +    // At this point, no more tasks are running for the stage (and the TaskSetManager considers the
    +    // stage complete), but the tasks that ran on HostA need to be re-run, so the DAGScheduler
    +    // should re-submit the stage.
    +    assert(taskSets.size === 2)
    +
    +    // Make sure that the stage that was re-submitted was the ShuffleMapStage (not the reduce
    +    // stage, which shouldn't be run until all of the tasks in the ShuffleMapStage complete on
    +    // alive executors).
    +    assert(taskSets(1).tasks(0).isInstanceOf[ShuffleMapTask])
    --- End diff --
    
    do you think its worth adding
    
    ```scala
    assert(taskSets(1).tasks.size === 1)
    ```
    here, to make sure that only the one task is resubmitted, not both?  If it weren't true, the test would fail later on anyway, but it might be helpful to get a more meaningful earlier error msg.  Not necessary, up to you on whether its worth adding.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    I merged this into master -- thanks for the review @squito.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73363/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72948/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72744/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72732/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72948/testReport)** for PR 16892 at commit [`f8407f2`](https://github.com/apache/spark/commit/f8407f22b8a763a83b3b8e91a7a0660829d655c1).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72744/testReport)** for PR 16892 at commit [`06b26d3`](https://github.com/apache/spark/commit/06b26d3738403da30bbf2c7ef2bfdb86039baa02).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #73363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73363/testReport)** for PR 16892 at commit [`1407b48`](https://github.com/apache/spark/commit/1407b4861e6fe3d178b82af46ba0408d8be51c82).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #73363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73363/testReport)** for PR 16892 at commit [`1407b48`](https://github.com/apache/spark/commit/1407b4861e6fe3d178b82af46ba0408d8be51c82).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/16892


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72958/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72958 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72958/testReport)** for PR 16892 at commit [`f8407f2`](https://github.com/apache/spark/commit/f8407f22b8a763a83b3b8e91a7a0660829d655c1).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/16892
  
    **[Test build #72958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72958/testReport)** for PR 16892 at commit [`f8407f2`](https://github.com/apache/spark/commit/f8407f22b8a763a83b3b8e91a7a0660829d655c1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #16892: [SPARK-19560] Improve DAGScheduler tests.

Posted by kayousterhout <gi...@git.apache.org>.
Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/16892#discussion_r102821513
  
    --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ---
    @@ -2031,6 +2051,11 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with Timeou
        * In this test, we run a map stage where one of the executors fails but we still receive a
        * "zombie" complete message from that executor. We want to make sure the stage is not reported
        * as done until all tasks have completed.
    +   *
    +   * Most of the functionality in this test is tested in "run trivial shuffle with out-of-band
    +   * executor failure and retry".  However, that test uses ShuffleMapStages that are followed by
    +   * a ResultStage, whereas in this test, the ShuffleMapStage is tested in isolation, without a
    +   * ResultStage after it.
    --- End diff --
    
    The original point of this test was to test a Map-only stage, so then it can't have a stage that follows it, right?  I thought your earlier comment ("That is testing directly submitting mapstage jobs. While a lot of the logic is the same, I think its worth its own test (even earlier versions of #16620 had the behavior wrong for these jobs)." was saying that it was important / useful to have this map-only test.  Let me know if that comment was based on the understanding that this tested multi-stage jobs and you think I should just remove this map-only test.
    
    I do agree that it would be useful to add another test that tests a job with more stages, which seems like it could reveal more bugs, but I'll hold off on doing that in this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org