You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by weixiuli <gi...@git.apache.org> on 2018/12/06 06:57:40 UTC
[GitHub] spark pull request #23243: add initRegisteredExecutorsDB
GitHub user weixiuli opened a pull request:
https://github.com/apache/spark/pull/23243
add initRegisteredExecutorsDB
## What changes were proposed in this pull request?
As we all know that spark on Yarn uses DB to record RegisteredExecutors information, when the ExternalShuffleService restart and it can be reload, which will be used as well .
While neither spark's standalone nor spark on k8s can record it's RegisteredExecutors information
by db or other ,so when ExternalShuffleService restart ,which RegisteredExecutors information will be lost,this is't what we want to.
This commit add initRegisteredExecutorsDB which can be used either spark standalone or spark on k8s to record RegisteredExecutors information , when the ExternalShuffleService restart and it can be reload, which will be used as well .
(Please fill in changes proposed in this fix)
## How was this patch tested?
test("test initRegisteredExecutorsDB ") {
val sparkConf = new SparkConf()
Utils.loadDefaultSparkProperties(sparkConf)
val securityManager = new SecurityManager(sparkConf)
sparkConf.set(config.SHUFFLE_SERVICE_DB_ENABLED.key, "true")
sparkConf.set(config.SHUFFLE_SERVICE_ENABLED.key, "true")
sparkConf.set("spark.local.dir", "/tmp")
val externalShuffleService = new ExternalShuffleService(sparkConf, securityManager)
}
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/weixiuli/spark branch-2.4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23243.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23243
----
commit 3591c16dc758b762ead253be490a67748c33078a
Author: 魏秀利 <we...@...>
Date: 2018-12-06T06:20:43Z
add initRegisteredExecutorsDB
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredE...
Posted by weixiuli <gi...@git.apache.org>.
Github user weixiuli commented on the issue:
https://github.com/apache/spark/pull/23243
@HyukjinKwon OK ,thank you!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredE...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/23243
cc @jiangxb1987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: add initRegisteredExecutorsDB
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23243
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: add initRegisteredExecutorsDB
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23243
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: [branch-2.4][ExternalShuffleService]add initRegisteredEx...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23243
Backport from which JIRA @weixiuli? Usually the fix should go to master first and it's backported to other branches when it's needed. If it should be fixed in master branch as well, let's file a JIRA and switch the branch to the master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: add initRegisteredExecutorsDB
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23243
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23243: [SPARK-26288][ExternalShuffleService]add initRegisteredE...
Posted by weixiuli <gi...@git.apache.org>.
Github user weixiuli commented on the issue:
https://github.com/apache/spark/pull/23243
cc @gatorsmile @HyukjinKwon @ptkool @cloud-fan Kindly review
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org