You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "MarkGaox (via GitHub)" <gi...@apache.org> on 2024/09/12 23:08:27 UTC

[PR] Support cluster level customized stoppable check [helix]

MarkGaox opened a new pull request, #2919:
URL: https://github.com/apache/helix/pull/2919

   ### Issues
   
   - [X] My PR addresses the following Helix issues and references them in the PR description:
   #2918
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI changes:
   
   * This PR supports the cluster level customized stoppable check.
   * When configuring the RESTConfig with an specific URI (e.g., "https://*:123"), users means to use the cluster level health check instead of the instance/partition level checks. 
   * The cluster level health check will pass `instances`, `toBeStoppedInstances`, and the `clusterID` to the health check endpoint and expect the user to return the response regarding whether those instances are stoppable and the reason of being not stoppable. 
   * Support Redirect of request if receiving status code of 302 with max retry of 3 times.
   * Following is an expected request + response:
   ```
        example url: POST http://<baseUrl>/clusterHealthStatus -d {
          "instances" : ["n1", "n3", "n9"],
          "to_be_stopped_instances": "["n2", "n4"]",
          "cluster_name": "cluster1",
          "<key>": "<value>"
        ...
        }
        example response:
        {
          "stoppableInstances": ["n1", "n3"],
          "nonStoppableInstancesWithReasons": {
          "n2": "reason1,reason2",
          "n4": "reason3"
        }
   ```
   
   
   
   ### Tests
   
   - [X] The following tests are written for this issue:
   
    ```
   mvn test -Dtest=TestInstancesAccessor,TestMaintenanceManagementService,TestInstanceValidationUtilInRest,TestPerInstanceAccessor,TestCustomRestClient -pl helix-rest && mvn test -Dtest=TestInstanceValidationUtil -pl helix-core
   ```
   
   ### Changes that Break Backward Compatibility (Optional)
   
   - My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
   
   (Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
   
   ### Documentation (Optional)
   
   - In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on code in PR #2919:
URL: https://github.com/apache/helix/pull/2919#discussion_r1767675273


##########
helix-rest/src/main/java/org/apache/helix/rest/client/CustomRestClientImpl.java:
##########
@@ -110,42 +152,102 @@ protected JsonNode getJsonObject(HttpResponse httpResponse) throws IOException {
     return OBJECT_MAPPER.readTree(str);
   }
 
-  private Map<String, Boolean> handleResponse(HttpResponse httpResponse,
-      JsonConverter jsonConverter) throws IOException {
+  private <T> Map<String, T> handleResponse(HttpResponse httpResponse,
+      JsonConverter<T> jsonConverter) throws IOException {
     int status = httpResponse.getStatusLine().getStatusCode();
     if (status == HttpStatus.SC_OK) {
       LOG.info("Expected HttpResponse statusCode: {}", HttpStatus.SC_OK);
       return jsonConverter.convert(getJsonObject(httpResponse));
     } else {
       // Ensure entity is fully consumed so stream is closed.
       EntityUtils.consumeQuietly(httpResponse.getEntity());
-      throw new ClientProtocolException("Unexpected response status: " + status + ", reason: "
-          + httpResponse.getStatusLine().getReasonPhrase());
+      throw new ClientProtocolException(
+          "Unexpected response status: " + status + ", reason: " + httpResponse.getStatusLine()
+              .getReasonPhrase());
     }
   }
 
   @VisibleForTesting
   protected HttpResponse post(String url, Map<String, Object> payloads) throws IOException {
     HttpPost postRequest = new HttpPost(url);
-    try {
-      postRequest.setHeader("Accept", ACCEPT_CONTENT_TYPE);
-      StringEntity entity = new StringEntity(OBJECT_MAPPER.writeValueAsString(payloads),
-          ContentType.APPLICATION_JSON);
-      postRequest.setEntity(entity);
-      LOG.info("Executing request: {}, headers: {}, entity: {}", postRequest.getRequestLine(),
-          postRequest.getAllHeaders(), postRequest.getEntity());
-
-      HttpResponse response = _httpClient.execute(postRequest);
-      int status = response.getStatusLine().getStatusCode();
-      if (status != HttpStatus.SC_OK) {
-        LOG.warn("Received non-200 status code: {}, payloads: {}", status, payloads);
+    int retries = 0;
+    HttpResponse response = null;
+
+    while (retries < MAX_REDIRECT) {
+      try {
+        postRequest.setHeader("Accept", ACCEPT_CONTENT_TYPE);
+        StringEntity entity = new StringEntity(OBJECT_MAPPER.writeValueAsString(payloads),
+            ContentType.APPLICATION_JSON);
+        postRequest.setEntity(entity);
+        LOG.info("Executing request: {}, headers: {}, entity: {}", postRequest.getRequestLine(),
+            postRequest.getAllHeaders(), postRequest.getEntity());
+
+        // Execute the request
+        response = _httpClient.execute(postRequest);
+        int status = response.getStatusLine().getStatusCode();
+
+        if (status == HttpStatus.SC_OK) {
+          return response;  // Return the successful response
+        } else if (status == HttpStatus.SC_MOVED_TEMPORARILY) {
+          // If receiving 302 Found (redirect), handle the redirection
+          Header locationHeader = response.getFirstHeader("Location");
+          if (locationHeader != null) {
+            String redirectUrl = locationHeader.getValue();
+            LOG.info("Redirecting to: {}", redirectUrl);
+            // Update the post request with the new URL
+            postRequest = new HttpPost(redirectUrl);
+            retries++;
+            LOG.info("Retrying redirect (attempt {} of {})", retries, MAX_REDIRECT);
+          } else {
+            LOG.warn("Received 302 but no Location header is present, stopping retries.");
+            break;  // Break out if there is no valid redirect location
+          }
+        } else {
+          LOG.warn("Received non-200 and non-302 status code: {}, payloads: {}", status, payloads);
+          return response;  // Return response without retry
+        }
+      } catch (IOException e) {
+        // Release connection to be reused and avoid connection leakage
+        postRequest.releaseConnection();
+        throw e;
       }
+    }
 
-      return response;
-    } catch (IOException e) {
-      // Release connection to be reused and avoid connection leakage.
-      postRequest.releaseConnection();
-      throw e;
+    if (retries >= MAX_REDIRECT) {
+      throw new IOException("Maximum redirect retries reached.");

Review Comment:
   Ya, it's probably not needed. We can just return the response in that case and the method `handleResponse` will generate the proper exception.
   Will remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on code in PR #2919:
URL: https://github.com/apache/helix/pull/2919#discussion_r1767678034


##########
helix-rest/src/main/java/org/apache/helix/rest/client/CustomRestClientImpl.java:
##########
@@ -93,15 +98,52 @@ public Map<String, Boolean> getPartitionStoppableCheck(String baseUrl, List<Stri
     if (partitions != null) {
       payLoads.put(PARTITIONS, partitions);
     }
-    JsonConverter jsonConverter = jsonNode -> {
-      Map<String, Boolean> result = new HashMap<>();
-      jsonNode.fields().forEachRemaining(
-          kv -> result.put(kv.getKey(), kv.getValue().get(IS_HEALTHY_FIELD).asBoolean()));
-      return result;
-    };
+    JsonConverter<Boolean> jsonConverter = this::extractPartitionBooleanMap;
     return handleResponse(post(url, payLoads), jsonConverter);
   }
 
+  @Override
+  public Map<String, List<String>> getAggregatedStoppableCheck(String baseUrl,
+      List<String> instances, Set<String> toBeStoppedInstances, String clusterId,

Review Comment:
   Thanks for catching it. It should use clusterId all the time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2919:
URL: https://github.com/apache/helix/pull/2919#discussion_r1767155296


##########
helix-rest/src/main/java/org/apache/helix/rest/client/CustomRestClientImpl.java:
##########
@@ -110,42 +152,102 @@ protected JsonNode getJsonObject(HttpResponse httpResponse) throws IOException {
     return OBJECT_MAPPER.readTree(str);
   }
 
-  private Map<String, Boolean> handleResponse(HttpResponse httpResponse,
-      JsonConverter jsonConverter) throws IOException {
+  private <T> Map<String, T> handleResponse(HttpResponse httpResponse,
+      JsonConverter<T> jsonConverter) throws IOException {
     int status = httpResponse.getStatusLine().getStatusCode();
     if (status == HttpStatus.SC_OK) {
       LOG.info("Expected HttpResponse statusCode: {}", HttpStatus.SC_OK);
       return jsonConverter.convert(getJsonObject(httpResponse));
     } else {
       // Ensure entity is fully consumed so stream is closed.
       EntityUtils.consumeQuietly(httpResponse.getEntity());
-      throw new ClientProtocolException("Unexpected response status: " + status + ", reason: "
-          + httpResponse.getStatusLine().getReasonPhrase());
+      throw new ClientProtocolException(
+          "Unexpected response status: " + status + ", reason: " + httpResponse.getStatusLine()
+              .getReasonPhrase());
     }
   }
 
   @VisibleForTesting
   protected HttpResponse post(String url, Map<String, Object> payloads) throws IOException {
     HttpPost postRequest = new HttpPost(url);
-    try {
-      postRequest.setHeader("Accept", ACCEPT_CONTENT_TYPE);
-      StringEntity entity = new StringEntity(OBJECT_MAPPER.writeValueAsString(payloads),
-          ContentType.APPLICATION_JSON);
-      postRequest.setEntity(entity);
-      LOG.info("Executing request: {}, headers: {}, entity: {}", postRequest.getRequestLine(),
-          postRequest.getAllHeaders(), postRequest.getEntity());
-
-      HttpResponse response = _httpClient.execute(postRequest);
-      int status = response.getStatusLine().getStatusCode();
-      if (status != HttpStatus.SC_OK) {
-        LOG.warn("Received non-200 status code: {}, payloads: {}", status, payloads);
+    int retries = 0;
+    HttpResponse response = null;
+
+    while (retries < MAX_REDIRECT) {
+      try {
+        postRequest.setHeader("Accept", ACCEPT_CONTENT_TYPE);
+        StringEntity entity = new StringEntity(OBJECT_MAPPER.writeValueAsString(payloads),
+            ContentType.APPLICATION_JSON);
+        postRequest.setEntity(entity);
+        LOG.info("Executing request: {}, headers: {}, entity: {}", postRequest.getRequestLine(),
+            postRequest.getAllHeaders(), postRequest.getEntity());
+
+        // Execute the request
+        response = _httpClient.execute(postRequest);
+        int status = response.getStatusLine().getStatusCode();
+
+        if (status == HttpStatus.SC_OK) {
+          return response;  // Return the successful response
+        } else if (status == HttpStatus.SC_MOVED_TEMPORARILY) {
+          // If receiving 302 Found (redirect), handle the redirection
+          Header locationHeader = response.getFirstHeader("Location");
+          if (locationHeader != null) {
+            String redirectUrl = locationHeader.getValue();
+            LOG.info("Redirecting to: {}", redirectUrl);
+            // Update the post request with the new URL
+            postRequest = new HttpPost(redirectUrl);
+            retries++;
+            LOG.info("Retrying redirect (attempt {} of {})", retries, MAX_REDIRECT);
+          } else {
+            LOG.warn("Received 302 but no Location header is present, stopping retries.");
+            break;  // Break out if there is no valid redirect location
+          }
+        } else {
+          LOG.warn("Received non-200 and non-302 status code: {}, payloads: {}", status, payloads);
+          return response;  // Return response without retry
+        }
+      } catch (IOException e) {
+        // Release connection to be reused and avoid connection leakage
+        postRequest.releaseConnection();
+        throw e;
       }
+    }
 
-      return response;
-    } catch (IOException e) {
-      // Release connection to be reused and avoid connection leakage.
-      postRequest.releaseConnection();
-      throw e;
+    if (retries >= MAX_REDIRECT) {
+      throw new IOException("Maximum redirect retries reached.");

Review Comment:
   Why throw exp here? 



##########
helix-core/src/main/java/org/apache/helix/model/RESTConfig.java:
##########
@@ -83,4 +85,18 @@ public String getBaseUrl(String instance) {
 
     return baseUrl.replace("*", instanceVip);
   }
+
+  /**
+   * Retrieves the configured health URL if no wildcard is present.

Review Comment:
   I would suggest include more detail about aggregated healthcheck url and individual health check url. What are the differences, example etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on PR #2919:
URL: https://github.com/apache/helix/pull/2919#issuecomment-2373418682

   > I don't have major concerns. But take a look of failure tests.
   The failed test passed locally. 
   
   <img width="350" alt="Screenshot 2024-09-25 at 1 30 42 AM" src="https://github.com/user-attachments/assets/d1aa98fb-056d-47b1-88d4-20e3d3986912">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "MarkGaox (via GitHub)" <gi...@apache.org>.
MarkGaox commented on PR #2919:
URL: https://github.com/apache/helix/pull/2919#issuecomment-2373420543

   This PR is approved by @zpinto and @junkaixue. TFTR. It's ready to be merged. The commit message: "Support aggregated endpoint for customized stoppable check"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on code in PR #2919:
URL: https://github.com/apache/helix/pull/2919#discussion_r1764027027


##########
helix-rest/src/main/java/org/apache/helix/rest/client/CustomRestClientImpl.java:
##########
@@ -93,15 +98,52 @@ public Map<String, Boolean> getPartitionStoppableCheck(String baseUrl, List<Stri
     if (partitions != null) {
       payLoads.put(PARTITIONS, partitions);
     }
-    JsonConverter jsonConverter = jsonNode -> {
-      Map<String, Boolean> result = new HashMap<>();
-      jsonNode.fields().forEachRemaining(
-          kv -> result.put(kv.getKey(), kv.getValue().get(IS_HEALTHY_FIELD).asBoolean()));
-      return result;
-    };
+    JsonConverter<Boolean> jsonConverter = this::extractPartitionBooleanMap;
     return handleResponse(post(url, payLoads), jsonConverter);
   }
 
+  @Override
+  public Map<String, List<String>> getAggregatedStoppableCheck(String baseUrl,
+      List<String> instances, Set<String> toBeStoppedInstances, String clusterId,

Review Comment:
   Why do we use clusterId in some places and clusterName in others?



##########
helix-core/src/main/java/org/apache/helix/model/RESTConfig.java:
##########
@@ -61,12 +63,12 @@ public String get(SimpleFields property) {
   }
 
   /**
-   * Get the base restful endpoint of the instance
+   * Resolves the customized health URL by replacing the wildcard with the instance's name
    *
    * @param instance The instance
    * @return The base restful endpoint
    */
-  public String getBaseUrl(String instance) {
+  public String resolveInstanceHealthUrl(String instance) {

Review Comment:
   This is backwards incompatible change to rename public method.



##########
helix-core/src/main/java/org/apache/helix/model/RESTConfig.java:
##########
@@ -83,4 +85,18 @@ public String getBaseUrl(String instance) {
 
     return baseUrl.replace("*", instanceVip);
   }
+
+  /**
+   * Retrieves the configured health URL if no wildcard is present.
+   * Returns an empty Optional if the URL contains a wildcard.
+   *
+   * @return Optional containing the exact configured URL, or empty if a wildcard is present.
+   */
+  public Optional<String> getConfiguredHealthUrl() {

Review Comment:
   Can we think of a more descriptive name for this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on PR #2919:
URL: https://github.com/apache/helix/pull/2919#issuecomment-2361530190

   Also could you please update the description with new naming? Thx 😄


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on code in PR #2919:
URL: https://github.com/apache/helix/pull/2919#discussion_r1765574754


##########
helix-core/src/main/java/org/apache/helix/model/RESTConfig.java:
##########
@@ -83,4 +85,18 @@ public String getBaseUrl(String instance) {
 
     return baseUrl.replace("*", instanceVip);
   }
+
+  /**
+   * Retrieves the configured health URL if no wildcard is present.
+   * Returns an empty Optional if the URL contains a wildcard.
+   *
+   * @return Optional containing the exact configured URL, or empty if a wildcard is present.
+   */
+  public Optional<String> getConfiguredHealthUrl() {

Review Comment:
   `getCompleteConfiguredHealthUrl`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support cluster level customized stoppable check [helix]

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on PR #2919:
URL: https://github.com/apache/helix/pull/2919#issuecomment-2351282891

   could you please also update naming in pr? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


Re: [PR] Support aggregated endpoint for customized stoppable check [helix]

Posted by "junkaixue (via GitHub)" <gi...@apache.org>.
junkaixue merged PR #2919:
URL: https://github.com/apache/helix/pull/2919


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org