You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by XuTingjun <gi...@git.apache.org> on 2015/09/14 10:16:33 UTC
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
GitHub user XuTingjun opened a pull request:
https://github.com/apache/spark/pull/8741
[SPARK-10586] fix bug: BlockManager ca't be removed when it is re-registered, then disassociats
Scene: When the executor has been removed, but it still exists on the SparkUI web.
Process:
1. Driver Lost executor because heartbeat timed out;
2. Executor received SIGNAL 15: SIGTERM and **re-registered** the blockmanager;
3. Driver Lost executor because remote Rpc client disassociated and attempt to remove blockmanager but not success;
Reason:
The first time lost executor, the [SchedulerBackend reviveOffers](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L481), when the host also has other executors, it won't [add this executor](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L288). So the second time lost executor, DAGScheduler won't remove BlockManager because the executorId has been in ```failedEpoch```.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/XuTingjun/spark SPARK-10586
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8741.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8741
----
commit bb562a269b954639a8f254d79782a2f7d0e209c9
Author: xutingjun <xu...@huawei.com>
Date: 2015-09-14T07:43:29Z
when executor adds, delete it from failedEpoch
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151593985
Hi @XuTingjun , can you add a test case for this, perhaps to `TaskSchedulerImplSuite`? If I understand correctly, the key issue is when you have two executors on one host, so that if you lose one and try to add it back the re-registration is wrong. But I'm not sure I follow the details -- a test case would really help clarify.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151109274
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8741
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-164619843
@XuTingjun I'm also not sure if I follow. If the driver loses an executor, it should just remove it from all of its schedulers' data structures. When the executor re-registers, it's a new executor according to the driver, so there's no problem. Actually, can you test out the latest code in master and verify whether this problem still exists?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151080536
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-139994290
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-140036565
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42416/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/8741#discussion_r43162744
--- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -285,9 +285,12 @@ private[spark] class TaskSchedulerImpl(
activeExecutorIds += o.executorId
if (!executorsByHost.contains(o.host)) {
executorsByHost(o.host) = new HashSet[String]()
+ }
+ if (!executorsByHost.get(o.host).get.contains(o.executorId)) {
--- End diff --
`!executorsByHost(o.host).contains(o.executorId)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151084189
**[Test build #44350 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44350/consoleFull)** for PR 8741 at commit [`bb562a2`](https://github.com/apache/spark/commit/bb562a269b954639a8f254d79782a2f7d0e209c9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151109164
**[Test build #44350 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44350/consoleFull)** for PR 8741 at commit [`bb562a2`](https://github.com/apache/spark/commit/bb562a269b954639a8f254d79782a2f7d0e209c9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151109276
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44350/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-139994224
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-140036563
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-151080618
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-139995696
[Test build #42416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42416/consoleFull) for PR 8741 at commit [`bb562a2`](https://github.com/apache/spark/commit/bb562a269b954639a8f254d79782a2f7d0e209c9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10586] fix bug: BlockManager ca't be re...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8741#issuecomment-140036476
[Test build #42416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42416/console) for PR 8741 at commit [`bb562a2`](https://github.com/apache/spark/commit/bb562a269b954639a8f254d79782a2f7d0e209c9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class Stddev(child: Expression) extends StddevAgg(child) `
* `case class StddevPop(child: Expression) extends StddevAgg(child) `
* `case class StddevSamp(child: Expression) extends StddevAgg(child) `
* `abstract class StddevAgg(child: Expression) extends AlgebraicAggregate `
* `abstract class StddevAgg1(child: Expression) extends UnaryExpression with PartialAggregate1 `
* `case class Stddev(child: Expression) extends StddevAgg1(child) `
* `case class StddevPop(child: Expression) extends StddevAgg1(child) `
* `case class StddevSamp(child: Expression) extends StddevAgg1(child) `
* `case class ComputePartialStd(child: Expression) extends UnaryExpression with AggregateExpression1 `
* `case class ComputePartialStdFunction (`
* `case class MergePartialStd(`
* `case class MergePartialStdFunction(`
* `case class StddevFunction(`
* `case class IntersectNode(conf: SQLConf, left: LocalNode, right: LocalNode)`
* `case class SampleNode(`
* `case class TakeOrderedAndProjectNode(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org