You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hustfxj <gi...@git.apache.org> on 2017/01/10 01:43:58 UTC
[GitHub] spark pull request #16525: the new leader will lost the statistics of the dr...
GitHub user hustfxj opened a pull request:
https://github.com/apache/spark/pull/16525
the new leader will lost the statistics of the driver's resource on the worker When the leader master has changed.
I deploy the standalone cluster with two masters. and utilize zooKeeper to provide leader election. Firstly, I submit the application with cluster mode. Then I kill the leader master, and the standby master will be the leader. But the new leader will lost the statistics of the driver's resource. Then I stop the application, we will see the negative used resource at the worker from masterPage. Like that:
```
Workers
Worker Id Address State Cores Memory
worker-20161220162751-10.125.6.222-59295 10.125.6.222:59295 ALIVE 4 (-1 Used) 6.8 GB (-1073741824.0 B Used)
worker-20161220164233-10.218.135.80-10944 10.218.135.80:10944 ALIVE 4 (0 Used) 6.8 GB (0.0 B Used)
```
Because the new leader forget calculate the driver\u2018 resource when the master receive the "WorkerLatestState" message. At the same time we can set RUNNING state for the app after the master receive the message\uff0c otherwise the app' state will still be WAITTING.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hustfxj/spark 2017
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16525.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16525
----
commit 764731714bfc2f07dcc5090b977af0e22301ea47
Author: xiaojian.fxj <xi...@alibaba-inc.com>
Date: 2017-01-10T01:42:35Z
the new leader will lost the statistics of the driver's resource on the worker When the leader master has changed.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16525: [SPARK-18959] the new leader will lost the statistics of...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16525
@hustfxj Unluckily we don't support multi-master nodes in standalone mode, so could you please close this PR? Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16525: [SPARK-18959] the new leader will lost the statistics of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16525
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16525: [SPARK-18959] the new leader will lost the statis...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16525
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16525: [SPARK-18959] the new leader will lost the statistics of...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16525
@srowen @andrewor14 can you review it again? Thank you
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16525: [SPARK-18959] the new leader will lost the statistics of...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16525
@srowen can you help review it ? I think it is a bug. Thank you very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org