You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/27 04:06:09 UTC
[GitHub] [spark] beliefer opened a new pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
beliefer opened a new pull request #28641:
URL: https://github.com/apache/spark/pull/28641
### What changes were proposed in this pull request?
`DAGSchedulerSuite `provides `completeShuffleMapStageSuccessfully `to make `ShuffleMapStage `successfully.
But many test case uses complete directly as follows:
`complete(taskSets(0), Seq((Success, makeMapStatus("hostA", 1))))`
We need to improve `completeShuffleMapStageSuccessfully `and reuse it.
`completeShuffleMapStageSuccessfully(0, 0, 1, Some(0), Seq("hostA"))`
### Why are the changes needed?
Improve and reuse completeShuffleMapStageSuccessfully
### Does this PR introduce _any_ user-facing change?
'No'.
### How was this patch tested?
Jenkins test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642721929
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643069579
**[Test build #123879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123879/testReport)** for PR 28641 at commit [`b14ac2d`](https://github.com/apache/spark/commit/b14ac2d5f40b0065da989ebc2fec289c0bb6be5a).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r435116220
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
Review comment:
Only two test cases passed `execIds `. I will delete this parameter too.
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
Review comment:
Only two test cases passed `execIds`. I will delete this parameter too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643070215
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123879/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642726571
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-634381384
cc @jiangxb1987 @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638735414
**[Test build #123526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123526/testReport)** for PR 28641 at commit [`27c937b`](https://github.com/apache/spark/commit/27c937bc18a108e040568a94666905f5497c16c7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643123638
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643249467
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-633922057
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642652240
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123845/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642651318
**[Test build #123845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123845/testReport)** for PR 28641 at commit [`03ff735`](https://github.com/apache/spark/commit/03ff73512c014ca6e087ea6f67f44b01f0e4f4ce).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644380608
**[Test build #124071 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124071/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r438949977
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +884,29 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param hostNames - Host on which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ hostNames: Seq[String] = Seq.empty[String]): Unit = {
+ def compareStageAttempt(taskSet: TaskSet): Boolean = {
+ taskSet.stageId == stageId && taskSet.stageAttemptId == attemptIdx
+ }
+
+ val stageAttemptOpt = taskSets.find(compareStageAttempt(_))
+ assert(stageAttemptOpt.isDefined)
+ val stageAttempt = stageAttemptOpt.get
checkStageId(stageId, attemptIdx, stageAttempt)
complete(stageAttempt, stageAttempt.tasks.zipWithIndex.map {
case (task, idx) =>
- (Success, makeMapStatus("host" + ('A' + idx).toChar, numShufflePartitions))
+ val hostName = if (idx < hostNames.size) {
+ hostNames(idx)
+ } else {
+ "host" + ('A' + idx).toChar
Review comment:
nit: s"host${('A' + 2).toChar}"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643105943
**[Test build #123880 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123880/testReport)** for PR 28641 at commit [`0a70316`](https://github.com/apache/spark/commit/0a70316f3127572f6061c32ad31ebb5dab462e12).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642378743
**[Test build #123809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123809/testReport)** for PR 28641 at commit [`8d16061`](https://github.com/apache/spark/commit/8d1606131162c1be0b1363120186a90b9f8c4914).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643070210
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644263709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644381786
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643041896
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642575154
**[Test build #123845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123845/testReport)** for PR 28641 at commit [`03ff735`](https://github.com/apache/spark/commit/03ff73512c014ca6e087ea6f67f44b01f0e4f4ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638808348
**[Test build #123526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123526/testReport)** for PR 28641 at commit [`27c937b`](https://github.com/apache/spark/commit/27c937bc18a108e040568a94666905f5497c16c7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643549980
**[Test build #123953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123953/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643123004
**[Test build #123903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123903/testReport)** for PR 28641 at commit [`0a70316`](https://github.com/apache/spark/commit/0a70316f3127572f6061c32ad31ebb5dab462e12).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643123638
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642726074
**[Test build #123856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123856/testReport)** for PR 28641 at commit [`03ff735`](https://github.com/apache/spark/commit/03ff73512c014ca6e087ea6f67f44b01f0e4f4ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638735948
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r435070559
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
Review comment:
@Ngone51 Thanks! Only two test cases passed maptaskids. I will delete this parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642430811
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643549852
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643249473
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123914/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642865309
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r439288152
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +883,29 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param hostNames - Host on which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ hostNames: Seq[String] = Seq.empty[String]): Unit = {
+ def compareStageAttempt(taskSet: TaskSet): Boolean = {
+ taskSet.stageId == stageId && taskSet.stageAttemptId == attemptIdx
+ }
+
+ val stageAttemptOpt = taskSets.find(compareStageAttempt(_))
+ assert(stageAttemptOpt.isDefined)
+ val stageAttempt = stageAttemptOpt.get
checkStageId(stageId, attemptIdx, stageAttempt)
Review comment:
Yeah!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643032785
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638735948
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642379023
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643120902
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643161089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642726571
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642575665
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-633922562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r439287499
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +883,29 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param hostNames - Host on which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ hostNames: Seq[String] = Seq.empty[String]): Unit = {
+ def compareStageAttempt(taskSet: TaskSet): Boolean = {
+ taskSet.stageId == stageId && taskSet.stageAttemptId == attemptIdx
+ }
+
+ val stageAttemptOpt = taskSets.find(compareStageAttempt(_))
+ assert(stageAttemptOpt.isDefined)
+ val stageAttempt = stageAttemptOpt.get
checkStageId(stageId, attemptIdx, stageAttempt)
Review comment:
nit: This check is no longer needed because we've checked it in `compareStageAttempt`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643215886
**[Test build #123903 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123903/testReport)** for PR 28641 at commit [`0a70316`](https://github.com/apache/spark/commit/0a70316f3127572f6061c32ad31ebb5dab462e12).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r433979634
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
Review comment:
Well, there's another concern that while we're reusing the function in more places, it gets more complicated to use because of new arguments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643249467
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642865309
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r438200288
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +884,27 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param hostNames - Host on which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ hostNames: Seq[String] = Seq.empty[String]): Unit = {
+ def compareStageAttempt(taskSet: TaskSet): Boolean = {
+ taskSet.stageId == stageId && taskSet.stageAttemptId == attemptIdx
+ }
+
+ val stageAttempt = taskSets.find(compareStageAttempt(_)).getOrElse(taskSets.last)
Review comment:
When will `getOrElse(taskSets.last)` happen?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638735414
**[Test build #123526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123526/testReport)** for PR 28641 at commit [`27c937b`](https://github.com/apache/spark/commit/27c937bc18a108e040568a94666905f5497c16c7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642652226
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-633922057
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642378743
**[Test build #123809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123809/testReport)** for PR 28641 at commit [`8d16061`](https://github.com/apache/spark/commit/8d1606131162c1be0b1363120186a90b9f8c4914).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644394716
Thanks, merged to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643572136
**[Test build #123953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123953/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642863862
**[Test build #123856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123856/testReport)** for PR 28641 at commit [`03ff735`](https://github.com/apache/spark/commit/03ff73512c014ca6e087ea6f67f44b01f0e4f4ce).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638809374
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644481669
@jiangxb1987 @Ngone51 Thanks for your help!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 closed pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
jiangxb1987 closed pull request #28641:
URL: https://github.com/apache/spark/pull/28641
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643032435
**[Test build #123879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123879/testReport)** for PR 28641 at commit [`b14ac2d`](https://github.com/apache/spark/commit/b14ac2d5f40b0065da989ebc2fec289c0bb6be5a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643041446
**[Test build #123880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123880/testReport)** for PR 28641 at commit [`0a70316`](https://github.com/apache/spark/commit/0a70316f3127572f6061c32ad31ebb5dab462e12).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643107717
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r439177584
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +884,29 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param hostNames - Host on which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ hostNames: Seq[String] = Seq.empty[String]): Unit = {
+ def compareStageAttempt(taskSet: TaskSet): Boolean = {
+ taskSet.stageId == stageId && taskSet.stageAttemptId == attemptIdx
+ }
+
+ val stageAttemptOpt = taskSets.find(compareStageAttempt(_))
+ assert(stageAttemptOpt.isDefined)
+ val stageAttempt = stageAttemptOpt.get
checkStageId(stageId, attemptIdx, stageAttempt)
complete(stageAttempt, stageAttempt.tasks.zipWithIndex.map {
case (task, idx) =>
- (Success, makeMapStatus("host" + ('A' + idx).toChar, numShufflePartitions))
+ val hostName = if (idx < hostNames.size) {
+ hostNames(idx)
+ } else {
+ "host" + ('A' + idx).toChar
Review comment:
OK. I will change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643217075
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643032785
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643161089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644263214
**[Test build #124071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124071/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642430256
**[Test build #123809 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123809/testReport)** for PR 28641 at commit [`8d16061`](https://github.com/apache/spark/commit/8d1606131162c1be0b1363120186a90b9f8c4914).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r433979634
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
Review comment:
Well, there's another concern that while we're reusing the function in more places, it gets more complicated to use after adding many new arguments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642430811
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644263214
**[Test build #124071 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124071/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644381786
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643217075
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643041446
**[Test build #123880 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123880/testReport)** for PR 28641 at commit [`0a70316`](https://github.com/apache/spark/commit/0a70316f3127572f6061c32ad31ebb5dab462e12).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642575665
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643160492
**[Test build #123914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123914/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r435119328
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ taskSetOpt: Option[Int] = None,
Review comment:
Good idea. I will improve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643572368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642430815
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123809/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644262390
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643550115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r438517228
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +884,27 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param hostNames - Host on which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ hostNames: Seq[String] = Seq.empty[String]): Unit = {
+ def compareStageAttempt(taskSet: TaskSet): Boolean = {
+ taskSet.stageId == stageId && taskSet.stageAttemptId == attemptIdx
+ }
+
+ val stageAttempt = taskSets.find(compareStageAttempt(_)).getOrElse(taskSets.last)
Review comment:
This indicates that the user has used the wrong `stageId` and `attemptIdx`.
Let us improve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642652226
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r435070559
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
Review comment:
@Ngone51 Thanks! Only two test cases passed `maptaskids`. I will delete this parameter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on a change in pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on a change in pull request #28641:
URL: https://github.com/apache/spark/pull/28641#discussion_r435012863
##########
File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
##########
@@ -889,16 +870,40 @@ class DAGSchedulerSuite extends SparkFunSuite with LocalSparkContext with TimeLi
* @param stageId - The current stageId
* @param attemptIdx - The current attempt count
* @param numShufflePartitions - The number of partitions in the next stage
+ * @param taskSetOpt - The index of task set
+ * @param hostNames - Host on which each task in the task set is executed
+ * @param mapTaskIds - ID of each task in the task set
+ * @param execIds - ID of the executor in which each task in the task set is executed
*/
private def completeShuffleMapStageSuccessfully(
stageId: Int,
attemptIdx: Int,
- numShufflePartitions: Int): Unit = {
- val stageAttempt = taskSets.last
+ numShufflePartitions: Int,
+ taskSetOpt: Option[Int] = None,
Review comment:
Is this TaskSet index really needed? We should avoid rely on the ordering of TaskSets to verify the results.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643217079
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123903/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643247249
**[Test build #123914 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123914/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643107717
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642726074
**[Test build #123856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123856/testReport)** for PR 28641 at commit [`03ff735`](https://github.com/apache/spark/commit/03ff73512c014ca6e087ea6f67f44b01f0e4f4ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-638809374
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643549980
**[Test build #123953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123953/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643107728
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123880/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643041896
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642575154
**[Test build #123845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123845/testReport)** for PR 28641 at commit [`03ff735`](https://github.com/apache/spark/commit/03ff73512c014ca6e087ea6f67f44b01f0e4f4ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-644263709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643572368
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643123004
**[Test build #123903 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123903/testReport)** for PR 28641 at commit [`0a70316`](https://github.com/apache/spark/commit/0a70316f3127572f6061c32ad31ebb5dab462e12).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643160492
**[Test build #123914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123914/testReport)** for PR 28641 at commit [`681d9e5`](https://github.com/apache/spark/commit/681d9e5d14800a171b97e6d73a02ae6f14eba19d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643550115
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-633922562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643070210
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-643032435
**[Test build #123879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123879/testReport)** for PR 28641 at commit [`b14ac2d`](https://github.com/apache/spark/commit/b14ac2d5f40b0065da989ebc2fec289c0bb6be5a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28641: [SPARK-31824][CORE][TESTS] DAGSchedulerSuite: Improve and reuse completeShuffleMapStageSuccessfully
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28641:
URL: https://github.com/apache/spark/pull/28641#issuecomment-642379023
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org