You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by suyanNone <gi...@git.apache.org> on 2015/09/01 11:19:59 UTC
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
GitHub user suyanNone opened a pull request:
https://github.com/apache/spark/pull/8550
[SPARK][SPARK-10370]Cancel all running attempts while that stage marked as finished
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/suyanNone/spark apache-tasksets
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8550.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8550
----
commit 75b97796a2226cbc855d8b6d198a09cce709d698
Author: hushan[胡珊] <hu...@xiaomi.com>
Date: 2015-09-01T09:16:15Z
Cancel stage task after it already marked finished
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136685203
[Test build #41867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41867/console) for PR 8550 at commit [`75b9779`](https://github.com/apache/spark/commit/75b97796a2226cbc855d8b6d198a09cce709d698).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by suyanNone <gi...@git.apache.org>.
Github user suyanNone closed the pull request at:
https://github.com/apache/spark/pull/8550
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136646923
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by suyanNone <gi...@git.apache.org>.
Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136766398
@squito I close that patch, because there have some errors in test.
call `taskScheduler.cancelTasks(stage.id, true)` in MarkStageAsFinished, It will mark all the no usefull tasksets related to that stage as zombie, because we already mark it as finished, and no need to deal with any running task for that stage. So first that prevent task from scheduling, second
it may be good to kill the running task for already finished stage, the task may run a long time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by suyanNone <gi...@git.apache.org>.
Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136767971
For Spark-2666, this patch is aim to cancel all stage tasks as long as a FetchFailedException was thrown, and I think is no related to this patch, right? because this patch is just cancel the task which his stage is marked as finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by suyanNone <gi...@git.apache.org>.
Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136762126
@squito yean, I will close that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136772133
@suyanNone ah, you are right, sorry I didn't see that `taskScheduler.cancelTasks` marks the `TaskSetManager` as zombie as well. so it would have the desired effect of marking the stage as zombie, but it has some strange side-effects, including logging that the stage has failed and sending a `TaskSetFailed` event to the dagscheduler, though in fact nothing has failed, the rest of the tasks are just no longer necessary.
That was the original purpose of SPARK-2666 ... I am proposing widening it a bit, to cover the interaction between "zombie" tasksets and cancelling tasks. I think that will lead to a net improvement in the scheduler code. But its possible they should just be completely separate issues, I think we need more input before deciding that.
Can we move this discussion to the jira for SPARK-2666? You have made some good points and they will most likely get lost on this pr ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136685255
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136647095
[Test build #41867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41867/consoleFull) for PR 8550 at commit [`75b9779`](https://github.com/apache/spark/commit/75b97796a2226cbc855d8b6d198a09cce709d698).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136761672
Hi @suyanNone thanks for submitting this, but this is actually not what that jira was referring to. We don't just need to cancel tasks that are already running -- we also need to mark the existing task set as a `zombie` in order to prevent even more tasks from getting scheduled. Canceling the running tasks would also be good, but it may be done as part of SPARK-2666, and there is still a bit of design discussion needed on that one. For SPARK-10370, I think we're blocked by SPARK-10372 to have a good way to test it.
Unfortunately these small changes look simple but are tough to reason about, so I think we want to proceed very cautiously. Do you mind closing this pr?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136685257
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41867/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK][SPARK-10370]Cancel all running attempt...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8550#issuecomment-136646939
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org