You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by guoxiaolongzte <gi...@git.apache.org> on 2017/05/03 08:57:23 UTC
[GitHub] spark pull request #17841: [SPARK-20577][DOC][CORE]Add REST API Documentatio...
GitHub user guoxiaolongzte opened a pull request:
https://github.com/apache/spark/pull/17841
[SPARK-20577][DOC][CORE]Add REST API Documentation in Cluster Mode.
## What changes were proposed in this pull request?
Add REST API Documentation in Cluster Mode.
Include:
Submit an application to the Master with parameters specified in the request.The request is assumed to be a [[SubmitRestProtocolRequest]] in the form of JSON.If the request is successfully processed, return an appropriate response to the client indicating so.
If a submission ID is specified in the URL, have the Master kill the corresponding
Driver and return an appropriate response to the client.
If a submission ID is specified in the URL, request the status of the corresponding
Driver from the Master and include it in the response.
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guoxiaolongzte/spark SPARK-20577
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17841.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17841
----
commit d383efba12c66addb17006dea107bb0421d50bc3
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-03-31T13:57:09Z
[SPARK-20177]Document about compression way has some little detail changes.
commit 3059013e9d2aec76def14eb314b6761bea0e7ca0
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-01T01:38:02Z
[SPARK-20177] event log add a space
commit 555cef88fe09134ac98fd0ad056121c7df2539aa
Author: guoxiaolongzte <gu...@zte.com.cn>
Date: 2017-04-02T00:16:08Z
'/applications/[app-id]/jobs' in rest api,status should be [running|succeeded|failed|unknown]
commit 46bb1ad3ddd9fb55b5607ac4f20213a90186cfe9
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T03:16:50Z
Merge branch 'master' of https://github.com/apache/spark into SPARK-20177
commit 0efb0dd9e404229cce638fe3fb0c966276784df7
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T03:47:53Z
[SPARK-20218]'/applications/[app-id]/stages' in REST API,add description.
commit 0e37fdeee28e31fc97436dabd001d3c85c5a7794
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-05T05:22:54Z
[SPARK-20218] '/applications/[app-id]/stages/[stage-id]' in REST API,remove redundant description.
commit 52641bb01e55b48bd9e8579fea217439d14c7dc7
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-07T06:24:58Z
Merge branch 'SPARK-20218'
commit d3977c9cab0722d279e3fae7aacbd4eb944c22f6
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-08T07:13:02Z
Merge branch 'master' of https://github.com/apache/spark
commit 137b90e5a85cde7e9b904b3e5ea0bb52518c4716
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-10T05:13:40Z
Merge branch 'master' of https://github.com/apache/spark
commit 0fe5865b8022aeacdb2d194699b990d8467f7a0a
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-10T10:25:22Z
Merge branch 'SPARK-20190' of https://github.com/guoxiaolongzte/spark
commit cf6f42ac84466960f2232c025b8faeb5d7378fe1
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-10T10:26:27Z
Merge branch 'master' of https://github.com/apache/spark
commit 685cd6b6e3799c7be65674b2670159ba725f0b8f
Author: 郭小龙 10207633 <gu...@zte.com.cn>
Date: 2017-04-14T01:12:41Z
Merge branch 'master' of https://github.com/apache/spark
commit c716a9231e9ab117d2b03ba67a1c8903d8d9da93
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-17T06:57:21Z
Merge branch 'master' of https://github.com/apache/spark
commit 679cec36a968fbf995b567ca5f6f8cbd8e32673f
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-19T07:20:08Z
Merge branch 'master' of https://github.com/apache/spark
commit 3c9387af84a8f39cf8c1ce19e15de99dfcaf0ca5
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-19T08:15:26Z
Merge branch 'master' of https://github.com/apache/spark
commit cb71f4462a0889cbb0843875b1e4cf14bcb0d020
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-20T05:52:06Z
Merge branch 'master' of https://github.com/apache/spark
commit ce92a7415a2026f5bf909820110a13750a0949e1
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-21T05:21:48Z
Merge branch 'master' of https://github.com/apache/spark
commit dd64342206041a8c3a282459e5f2b898dc558d89
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-21T08:44:25Z
Merge branch 'master' of https://github.com/apache/spark
commit bffd2bd00c6b0e20313756e133adca4c97707c67
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-28T01:36:29Z
Merge branch 'master' of https://github.com/apache/spark
commit 588d42a382345a071532ace1eab5457911f6aa46
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-04-28T05:02:36Z
Merge branch 'master' of https://github.com/apache/spark
commit 4bbeee1231275d1afa0775dbb61fcc5817f6e57c
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-05-02T02:30:52Z
Merge branch 'master' of https://github.com/apache/spark
commit 362e5ad12bfe013a7780d81b5067c2ff644efa05
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-05-03T06:47:54Z
Merge branch 'master' of https://github.com/apache/spark
commit a2ff3ef457e828269186d24f02e2fc8ae4a1e81a
Author: guoxiaolong <gu...@zte.com.cn>
Date: 2017-05-03T08:42:52Z
[SPARK-20577]Add REST API Documentation in Cluster Mode
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17841
@srowen
I'm sorry, I did not understand what you meant. Can you give me a detailed analysis? Thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17841
Nobody is suggesting it doesn't work. It is at least only known to work to support internal callers. It is subject to changes in the API. A public API in contrast needs to retain compatibility over longer periods. This is not something it seems the project wants to guarantee. Only public APIs need docs and this isn't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17841
In fact, the three api is three servlets, submitRequestServlet, killRequestServlet, statusRequestServlet. Theoretically should be open to enhance the function of spark.
As for why not open, I still difficult to understand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17841
Where is the risk? High concurrency? Transaction processing? This is where I am puzzled.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17841
I think @srowen already clarified it very clearly, you can use it at your own risk, but to make it public and add to the doc should be well considered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17841
@jerryshao
Yes, we now have the need to use REST APIs to submit SPARK jobs.
We submit the task of submitting a application to a WEB system, which is designed to manage the functions of the submitted job and set specific parameters and related configurations for different jobs. The WEB management system interacts with the spark cluster. The REST interface is The most convenient.
We have now used these REST APIS to meet our needs. But the current lack of official documents related description, I have learned through their own research and research to understand these interfaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17841: [SPARK-20577][DOC][CORE]Add REST API Documentatio...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte closed the pull request at:
https://github.com/apache/spark/pull/17841
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17841
@jerryshao
I said that the use of the scene is real, do you agree?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17841
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17841
Unless somebody says this isn't a private API, let's close this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17841
He is saying you are not intended to call this API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17841
It is not documented because it is not something you should call. You do so at your own risk. Documentation is appropriate for supported APIs only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by guoxiaolongzte <gi...@git.apache.org>.
Github user guoxiaolongzte commented on the issue:
https://github.com/apache/spark/pull/17841
ok,thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17841: [SPARK-20577][DOC][CORE]Add REST API Documentation in Cl...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/17841
I remember these REST APIs are not public APIs, it is only used for SparkSubmit internally. shall we add the docs about them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org