You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hustfxj <gi...@git.apache.org> on 2017/01/02 14:16:22 UTC
[GitHub] spark pull request #16450: [SPARK-19052] Bug fix: the rest api don't support...
GitHub user hustfxj opened a pull request:
https://github.com/apache/spark/pull/16450
[SPARK-19052] Bug fix: the rest api don't support multiple standby masters on standalone cluster
The driver only know a master's address which come from StandaloneRestServer's masterUrl If submitting the job by rest api.So we should give priority to set "spark.master" by "sparkProperties".
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hustfxj/spark apache-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16450.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16450
----
commit fbea26b4be507ead6bf80ae665b75113b5caaa68
Author: xiaojian.fxj <xi...@alibaba-inc.com>
Date: 2017-01-02T13:57:21Z
Bug fix: the rest server don't support multiple standby masters.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16450
I'm not sure this is a good change, because it means you don't override defaults set somewhere in a config file anymore. That is I'm not clear this isn't on purpose -- could be right, wrong, just not sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16450
I checked the other places, and I'm sure the other places is set "spark.master" by user configuration but not the master's address directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16450
If we submit the application submission by SPARK REST API, we transmit the configure by the sparkProperties. Like that:
```
curl -X POST http://spark-cluster-ip:6066/v1/submissions/create --header
"Content-Type:application/json;charset=UTF-8" --data'{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "myAppArgument1" ],
"appResource" : "file:/myfilepath/spark-job-1.0.jar",
"clientSparkVersion" : "2.1.0",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "com.mycompany.MyJob",
"sparkProperties" : {
"spark.jars" : "file:/myfilepath/spark-job-1.0.jar",
"spark.driver.supervise" : "true",
"spark.app.name" : "MyJob",
"spark.eventLog.enabled": "true",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://10.20.23.22:7077,10.20.23.21:7077"
}}
```
We hope "spark.master" of the driver 's configure should be "spark://10.20.23.22:7077,10.20.23.21:7077", but in fact it maybe spark://10.20.23.22:7077 due to the spark core's code\uff1a
```
val conf = new SparkConf(false)
.setAll(sparkProperties)
.set("spark.master", masterUrl)
```
If we kill the master(spark://10.20.23.22:7077), and the "spark://10.20.23.21:7077" will be master\u3002After we kill the driver, then the spark can't restart the driver automatically. Because the driver only know old master's address which is spark://10.20.23.22:7077.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16450
@srowen can you help review it ? I think it is a bug. Thank you very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16450
@hustfxj Unluckily we don't support multi-master nodes in standalone mode, so could you please close this PR? Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16450: [SPARK-19052] the restSubmissionServer don't supp...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16450
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16450
@srowen @andrewor14 can you see it again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16450
Where? I don't see any other instances of this pattern. It's easier if you add specifics when discussing code changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by hustfxj <gi...@git.apache.org>.
Github user hustfxj commented on the issue:
https://github.com/apache/spark/pull/16450
@srowen masterUrl maybe defaults set, but the user's program's configure should have high priority. The user's program's configure can be delivered by the REST API's params "sparkProperties".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] the restSubmissionServer don't support mul...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/16450
Aha that's making more sense to me. I don't know this code much. @andrewor14 would be an ideal reviewer but not sure if he's available at this point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16450: [SPARK-19052] Bug fix: the rest api don't support multip...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16450
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org