You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "xianjingfeng (via GitHub)" <gi...@apache.org> on 2023/03/04 08:36:50 UTC
[GitHub] [incubator-uniffle] xianjingfeng opened a new pull request, #684: [#80][Part-2] feat: add REST API for decommisson
xianjingfeng opened a new pull request, #684:
URL: https://github.com/apache/incubator-uniffle/pull/684
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[#123] feat(operator): support xxx"
- "[#233] fix: check null before access result in xxx"
- "[MINOR] refactor: fix typo in variable name"
- "[MINOR] docs: fix typo in README"
- "[#255] test: fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
Add REST API for decommisson
### Why are the changes needed?
Support shuffle server decommission. It is a part of #80
### Does this PR introduce _any_ user-facing change?
Env:
Server IP: 127.0.0.1
HTTP port: 19998
RPC port: 19999
Decommission example:
curl -XPOST -H "Content-type:application/json" "http://127.0.0.1:19998/api/server/decommission" -d '{"serverId:": "127.0.0.1:19999"}'
Cancel decommission example:
curl -XPOST -H "Content-type:application/json" "http://127.0.0.1:19998/api/server/cancelDecommission" -d '{"serverId:": "127.0.0.1:19999"}'
### How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1125665019
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
How do you think to struct REST api as follows:
```
GET /api/v1/servers # list all servers.
GET /api/v1/servers/:id # get the specific server with id.
POST /api/v1/servers/:id/decommission # start decommission of server with id
POST /api/v1/servers/:id/cancelDecommission #...
POST /api/v1/servers/decommission # batch decommission. similar to the current impl
POST /api/v1/servers/cancelDecommission # batch cancel decommission.
```
##########
common/src/test/java/org/apache/uniffle/common/metrics/TestUtils.java:
##########
@@ -42,4 +43,22 @@ public static String httpGetMetrics(String urlString) throws IOException {
in.close();
return content.toString();
}
+
+ public static String httpPost(String urlString, String postData) throws IOException {
+ URL url = new URL(urlString);
+ HttpURLConnection con = (HttpURLConnection) url.openConnection();
+ con.setDoOutput(true);
+ con.setRequestMethod("POST");
+ OutputStream outputStream = con.getOutputStream();
+ outputStream.write(postData.getBytes());
+ BufferedReader in = new BufferedReader(
+ new InputStreamReader(con.getInputStream()));
+ String inputLine;
+ StringBuffer content = new StringBuffer();
+ while ((inputLine = in.readLine()) != null) {
+ content.append(inputLine);
+ }
+ in.close();
Review Comment:
Could you use a try with resource here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #684: [#80][Part-2] feat: add REST API for decommisson
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1454666701
# [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#684](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ea37b61) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/313ddd69723e0c45cd4ddf271b08f6f94c4a723b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (313ddd6) will **increase** coverage by `0.84%`.
> The diff coverage is `23.28%`.
```diff
@@ Coverage Diff @@
## master #684 +/- ##
============================================
+ Coverage 60.79% 61.64% +0.84%
+ Complexity 1839 1750 -89
============================================
Files 221 203 -18
Lines 12655 10194 -2461
Branches 1069 1027 -42
============================================
- Hits 7694 6284 -1410
+ Misses 4553 3577 -976
+ Partials 408 333 -75
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/uniffle/coordinator/web/Response.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivd2ViL1Jlc3BvbnNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...e/uniffle/coordinator/web/servlet/BaseServlet.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivd2ViL3NlcnZsZXQvQmFzZVNlcnZsZXQuamF2YQ==) | `10.00% <10.00%> (ø)` | |
| [.../uniffle/coordinator/web/servlet/NodesServlet.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivd2ViL3NlcnZsZXQvTm9kZXNTZXJ2bGV0LmphdmE=) | `30.00% <30.00%> (ø)` | |
| [...dinator/web/servlet/CancelDecommissionServlet.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivd2ViL3NlcnZsZXQvQ2FuY2VsRGVjb21taXNzaW9uU2VydmxldC5qYXZh) | `33.33% <33.33%> (ø)` | |
| [...e/coordinator/web/servlet/DecommissionServlet.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivd2ViL3NlcnZsZXQvRGVjb21taXNzaW9uU2VydmxldC5qYXZh) | `33.33% <33.33%> (ø)` | |
| [.../apache/uniffle/coordinator/CoordinatorServer.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3IvQ29vcmRpbmF0b3JTZXJ2ZXIuamF2YQ==) | `59.37% <100.00%> (+1.99%)` | :arrow_up: |
| [.../java/org/apache/spark/shuffle/RssSparkConfig.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50LXNwYXJrL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc3Bhcmsvc2h1ZmZsZS9Sc3NTcGFya0NvbmZpZy5qYXZh) | | |
| [...ava/org/apache/spark/shuffle/RssShuffleHandle.java](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y2xpZW50LXNwYXJrL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc3Bhcmsvc2h1ZmZsZS9Sc3NTaHVmZmxlSGFuZGxlLmphdmE=) | | |
| [deploy/kubernetes/operator/pkg/utils/config.go](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3V0aWxzL2NvbmZpZy5nbw==) | | |
| [...y/kubernetes/operator/pkg/webhook/inspector/pod.go](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGVwbG95L2t1YmVybmV0ZXMvb3BlcmF0b3IvcGtnL3dlYmhvb2svaW5zcGVjdG9yL3BvZC5nbw==) | | |
| ... and [19 more](https://codecov.io/gh/apache/incubator-uniffle/pull/684?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1458187181
> @xianjingfeng There is a flaky test. https://github.com/apache/incubator-uniffle/actions/runs/4335961089/jobs/7570925520
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1454729648
> From the example listed in description, I have questions
>
> 1. Do we support decommission multiple server in one request?
> 2. Can we list all decommissioning and decommissioned servers?
Updated. Now supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1128021898
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
I'm not familiar with Jersey. Is it lightweight enough? Is it used widely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1130522796
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/servlet/BaseServlet.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.servlet;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.Callable;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.uniffle.coordinator.web.Response;
+
+public abstract class BaseServlet extends HttpServlet {
+ public static final String JSON_MIME_TYPE = "application/json";
+ final ObjectMapper mapper = new ObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handleGet(req, resp)));
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handlePost(req, resp)));
+ }
+
+ private Response handlerRequest(
+ Callable<Response> function) {
+ Response response;
+ try {
+ // todo: Do something for authentication
+ response = function.call();
+ } catch (Exception e) {
+ response = Response.fail(e.getMessage());
+ }
+ return response;
+ }
+
+ protected Response handleGet(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected Response handlePost(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected void writeJSON(final HttpServletResponse resp, final Object obj)
+ throws IOException {
+ if (obj == null) {
+ return;
+ }
+ resp.setContentType(JSON_MIME_TYPE);
+ final OutputStream stream = resp.getOutputStream();
+ mapper.writeValue(stream, obj);
+ }
+
+ protected <T> T parseParamsFromJson(HttpServletRequest req, Class<T> clazz) throws IOException {
Review Comment:
Looks like you change the serverIds to set..
Would you mind add a UT for this method? I'm not sure a list of string could automatically parsed as Set<String>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1458309236
> Env: Server IP: 127.0.0.1 HTTP port: 19998 RPC port: 19999
>
> Decommission example: curl -XPOST -H "Content-type:application/json" "http://127.0.0.1:19998/api/server/decommission" -d '{"serverIds:": ["127.0.0.1:19999"]}'
> Cancel decommission example: curl -XPOST -H "Content-type:application/json" "http://127.0.0.1:19998/api/server/cancelDecommission" -d '{"serverIds:": ["127.0.0.1:19999"]}'
> Get server list: /api/server/nodes[?id={serverId}][?status={serverStatus}] curl "http://127.0.0.1:19998/api/server/nodes?status=DECOMMISSIONING" curl "http://127.0.0.1:19998/api/server/nodes?status=ACTIVE"
Thanks @xianjingfeng for the work. I suggest adding some docs for this, in this PR or part-4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1130541088
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/servlet/BaseServlet.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.servlet;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.Callable;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.uniffle.coordinator.web.Response;
+
+public abstract class BaseServlet extends HttpServlet {
+ public static final String JSON_MIME_TYPE = "application/json";
+ final ObjectMapper mapper = new ObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handleGet(req, resp)));
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handlePost(req, resp)));
+ }
+
+ private Response handlerRequest(
+ Callable<Response> function) {
+ Response response;
+ try {
+ // todo: Do something for authentication
+ response = function.call();
+ } catch (Exception e) {
+ response = Response.fail(e.getMessage());
+ }
+ return response;
+ }
+
+ protected Response handleGet(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected Response handlePost(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected void writeJSON(final HttpServletResponse resp, final Object obj)
+ throws IOException {
+ if (obj == null) {
+ return;
+ }
+ resp.setContentType(JSON_MIME_TYPE);
+ final OutputStream stream = resp.getOutputStream();
+ mapper.writeValue(stream, obj);
+ }
+
+ protected <T> T parseParamsFromJson(HttpServletRequest req, Class<T> clazz) throws IOException {
Review Comment:
If not. the current UT will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129623096
##########
docs/coordinator_guide.md:
##########
@@ -130,3 +130,61 @@ PrometheusPushGatewayMetricReporter is one of the built-in metrics reporter, whi
|rss.metrics.prometheus.pushgateway.groupingkey|-| Specifies the grouping key which is the group and global labels of all metrics. The label name and value are separated by '=', and labels are separated by ';', e.g., k1=v1;k2=v2. Please ensure that your grouping key meets the [Prometheus requirements](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). |
|rss.metrics.prometheus.pushgateway.jobname|-| The job name under which metrics will be pushed. |
|rss.metrics.prometheus.pushgateway.report.interval.seconds|10| The interval in seconds for the reporter to report metrics. |
+
+## RESTful API
+
+### Fetch Shuffle servers
+
+<details>
+ <summary><code>GET</code> <code><b>/api/server/nodes</b></code> </summary>
+
+##### Parameters
+
+> |name|type|data type|description|
+> |----|----|---------|-----------|
+> |id|required|string|shuffle server id, eg:127.0.0.1:19999|
+> |status|optional|string|Shuffle server status, eg:ACTIVE, DECOMMISSIONING, DECOMMISSIONED|
+
+##### Example cURL
+
+> ```javascript
Review Comment:
Its color is richer and it looks more beautiful. Bash is ok for me too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1454709431
From the example listed in description, I have questions
1. Do we support decommission multiple server in one request?
2. Can we list all decommissioning and decommissioned servers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1126039877
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
It is troublesome to get parameters from path in the current framework. Should we introduce other REST frameworks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1460169080
> > Env: Server IP: 127.0.0.1 HTTP port: 19998 RPC port: 19999
> > Decommission example: curl -XPOST -H "Content-type:application/json" "http://127.0.0.1:19998/api/server/decommission" -d '{"serverIds:": ["127.0.0.1:19999"]}'
> > Cancel decommission example: curl -XPOST -H "Content-type:application/json" "http://127.0.0.1:19998/api/server/cancelDecommission" -d '{"serverIds:": ["127.0.0.1:19999"]}'
> > Get server list: /api/server/nodes[?id={serverId}][?status={serverStatus}] curl "http://127.0.0.1:19998/api/server/nodes?status=DECOMMISSIONING" curl "http://127.0.0.1:19998/api/server/nodes?status=ACTIVE"
>
> Thanks @xianjingfeng for the work. I suggest adding some docs for this, in this PR or part-4.
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1463416129
Thanks @xianjingfeng for the work, and thanks everyone reviewing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen merged pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen merged PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129018094
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
I'm ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129443279
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
+ new NodesServlet(this),
+ "/api/server/nodes");
+ jettyServer.addServlet(
+ new DecommissionServlet(this),
+ "/api/server/decommission");
+ jettyServer.addServlet(
+ new CancelDecommissionServlet(this),
+ "/api/server/cancelDecommission");
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1126063935
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
> Should we introduce other REST frameworks?
If it's too much trouble, I'm OK with current route path. If it's simple, I prefer the above REST path, which is more idiomatic. It's up to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] smallzhongfeng commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1461356979
The GA failure cause by CoordinatorMetricsTest, could you rebase the latest code and trigger test again ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1130539557
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/servlet/BaseServlet.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.servlet;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.Callable;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.uniffle.coordinator.web.Response;
+
+public abstract class BaseServlet extends HttpServlet {
+ public static final String JSON_MIME_TYPE = "application/json";
+ final ObjectMapper mapper = new ObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handleGet(req, resp)));
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handlePost(req, resp)));
+ }
+
+ private Response handlerRequest(
+ Callable<Response> function) {
+ Response response;
+ try {
+ // todo: Do something for authentication
+ response = function.call();
+ } catch (Exception e) {
+ response = Response.fail(e.getMessage());
+ }
+ return response;
+ }
+
+ protected Response handleGet(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected Response handlePost(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected void writeJSON(final HttpServletResponse resp, final Object obj)
+ throws IOException {
+ if (obj == null) {
+ return;
+ }
+ resp.setContentType(JSON_MIME_TYPE);
+ final OutputStream stream = resp.getOutputStream();
+ mapper.writeValue(stream, obj);
+ }
+
+ protected <T> T parseParamsFromJson(HttpServletRequest req, Class<T> clazz) throws IOException {
Review Comment:
I have tried, It will be parsed as `HashSet` automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1130541088
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/servlet/BaseServlet.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.servlet;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.Callable;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.uniffle.coordinator.web.Response;
+
+public abstract class BaseServlet extends HttpServlet {
+ public static final String JSON_MIME_TYPE = "application/json";
+ final ObjectMapper mapper = new ObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handleGet(req, resp)));
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handlePost(req, resp)));
+ }
+
+ private Response handlerRequest(
+ Callable<Response> function) {
+ Response response;
+ try {
+ // todo: Do something for authentication
+ response = function.call();
+ } catch (Exception e) {
+ response = Response.fail(e.getMessage());
+ }
+ return response;
+ }
+
+ protected Response handleGet(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected Response handlePost(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+ protected void writeJSON(final HttpServletResponse resp, final Object obj)
+ throws IOException {
+ if (obj == null) {
+ return;
+ }
+ resp.setContentType(JSON_MIME_TYPE);
+ final OutputStream stream = resp.getOutputStream();
+ mapper.writeValue(stream, obj);
+ }
+
+ protected <T> T parseParamsFromJson(HttpServletRequest req, Class<T> clazz) throws IOException {
Review Comment:
If not the current UT will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1127879763
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
There will be more and more rest apis in the future. It is still troublesome to add an api under the current framework. So I think we can consider it later. How about introduce Jersey? @advancedxy @jerqi @zuston @kaijchen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1128946130
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
+ new NodesServlet(this),
+ "/api/server/nodes");
+ jettyServer.addServlet(
+ new DecommissionServlet(this),
+ "/api/server/decommission");
+ jettyServer.addServlet(
+ new CancelDecommissionServlet(this),
+ "/api/server/cancelDecommission");
Review Comment:
Please fix the indentation here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129128722
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
I don't know if `Jersey` is lightweight. I hope the restful framework could be modern and lightweight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129275026
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
> There will be more and more rest apis in the future. It is still troublesome to add an api under the current framework. So I think we can consider it later. How about introduce Jersey? @advancedxy @jerqi @zuston @kaijchen
I'm ok with current status. Let's make the new REST server framework a followup PR. Jersey itself seems a bit complex, but the path annotation https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest/user-guide.html#d0e2081 is quite elegant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] smallzhongfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "smallzhongfeng (via GitHub)" <gi...@apache.org>.
smallzhongfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129050595
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/request/DecommissionRequest.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.request;
+
+import java.util.List;
+
+public class DecommissionRequest {
+ private List<String> serverIds;
Review Comment:
What about `List` -> `Set` ? It seems that there is no logic to handle duplicate serverId.
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/request/CancelDecommissionRequest.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.request;
+
+import java.util.List;
+
+public class CancelDecommissionRequest {
+ private List<String> serverIds;
Review Comment:
What about List -> Set ? It seems that there is no logic to handle duplicate serverId.
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/request/DecommissionRequest.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.request;
+
+import java.util.List;
+
+public class DecommissionRequest {
+ private List<String> serverIds;
+
+ public List<String> getServerIds() {
+ return serverIds;
+ }
+
+ public void setServerIds(List<String> serverIds) {
+ this.serverIds = serverIds;
+ }
+
Review Comment:
Remove blank lines
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/web/servlet/BaseServlet.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.coordinator.web.servlet;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.Callable;
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import com.fasterxml.jackson.annotation.JsonInclude;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.apache.uniffle.coordinator.web.Response;
+
+public abstract class BaseServlet extends HttpServlet {
+ public static final String JSON_MIME_TYPE = "application/json";
+ final ObjectMapper mapper = new ObjectMapper().setSerializationInclusion(JsonInclude.Include.NON_NULL);
+
+ @Override
+ protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handleGet(req, resp)));
+ }
+
+ @Override
+ protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException {
+ writeJSON(resp, handlerRequest(() -> handlePost(req, resp)));
+ }
+
+ private Response handlerRequest(
+ Callable<Response> function) {
+ Response response;
+ try {
+ // todo: Do something for authentication
+ response = function.call();
+ } catch (Exception e) {
+ response = Response.fail(e.getMessage());
+ }
+ return response;
+ }
+
+ protected Response handleGet(
+ HttpServletRequest req,
+ HttpServletResponse resp) throws ServletException, IOException {
+ throw new IOException("Method not support!");
+ }
+
+
Review Comment:
Remove blank lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1459611076
@smallzhongfeng Could you help me review this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129294862
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
Maybe we can refer to Apache Livy, it use `scalatra`. We can use similar Java framework.
https://www.51cto.com/article/311366.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#issuecomment-1455106744
@xianjingfeng There is a flaky test. https://github.com/apache/incubator-uniffle/actions/runs/4335961089/jobs/7570925520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1128945170
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/CoordinatorServer.java:
##########
@@ -179,6 +183,19 @@ private void initialization() throws Exception {
server = coordinatorFactory.getServer();
}
+ private void registerRESTAPI() throws Exception {
+ LOG.info("Register REST API");
+ jettyServer.addServlet(
Review Comment:
> I'm not familiar with Jersey. Is it lightweight enough? Is it used widely?
Yes. Hadoop and Spark use Jersey either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129470057
##########
docs/coordinator_guide.md:
##########
@@ -130,3 +130,61 @@ PrometheusPushGatewayMetricReporter is one of the built-in metrics reporter, whi
|rss.metrics.prometheus.pushgateway.groupingkey|-| Specifies the grouping key which is the group and global labels of all metrics. The label name and value are separated by '=', and labels are separated by ';', e.g., k1=v1;k2=v2. Please ensure that your grouping key meets the [Prometheus requirements](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). |
|rss.metrics.prometheus.pushgateway.jobname|-| The job name under which metrics will be pushed. |
|rss.metrics.prometheus.pushgateway.report.interval.seconds|10| The interval in seconds for the reporter to report metrics. |
+
+## RESTful API
+
+### Fetch Shuffle servers
Review Comment:
👍 I like the style here, The RESTful API section looks pretty on the Github page.
##########
docs/coordinator_guide.md:
##########
@@ -130,3 +130,61 @@ PrometheusPushGatewayMetricReporter is one of the built-in metrics reporter, whi
|rss.metrics.prometheus.pushgateway.groupingkey|-| Specifies the grouping key which is the group and global labels of all metrics. The label name and value are separated by '=', and labels are separated by ';', e.g., k1=v1;k2=v2. Please ensure that your grouping key meets the [Prometheus requirements](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). |
|rss.metrics.prometheus.pushgateway.jobname|-| The job name under which metrics will be pushed. |
|rss.metrics.prometheus.pushgateway.report.interval.seconds|10| The interval in seconds for the reporter to report metrics. |
+
+## RESTful API
+
+### Fetch Shuffle servers
+
+<details>
+ <summary><code>GET</code> <code><b>/api/server/nodes</b></code> </summary>
+
+##### Parameters
+
+> |name|type|data type|description|
+> |----|----|---------|-----------|
+> |id|required|string|shuffle server id, eg:127.0.0.1:19999|
+> |status|optional|string|Shuffle server status, eg:ACTIVE, DECOMMISSIONING, DECOMMISSIONED|
+
+##### Example cURL
+
+> ```javascript
Review Comment:
why use javascript here? Bash may be more appropriate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #684: [#80][Part-3] feat: add REST API for decommisson
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #684:
URL: https://github.com/apache/incubator-uniffle/pull/684#discussion_r1129477032
##########
docs/coordinator_guide.md:
##########
@@ -130,3 +130,61 @@ PrometheusPushGatewayMetricReporter is one of the built-in metrics reporter, whi
|rss.metrics.prometheus.pushgateway.groupingkey|-| Specifies the grouping key which is the group and global labels of all metrics. The label name and value are separated by '=', and labels are separated by ';', e.g., k1=v1;k2=v2. Please ensure that your grouping key meets the [Prometheus requirements](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). |
|rss.metrics.prometheus.pushgateway.jobname|-| The job name under which metrics will be pushed. |
|rss.metrics.prometheus.pushgateway.report.interval.seconds|10| The interval in seconds for the reporter to report metrics. |
+
+## RESTful API
Review Comment:
It just hit me that we are going to change a new RESTful web framework, how about add `beta` here
Something like `## RESTful API(beta)`.
I think we don't commit the RESTful API compatibility here until we settled down a better lightweight and modern RESTful web framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org