You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/05/19 23:09:31 UTC

[GitHub] [helix] xyuanlu opened a new pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

xyuanlu opened a new pull request #1747:
URL: https://github.com/apache/helix/pull/1747


   ### Issues
   
   - [X] My PR addresses the following Helix issues and references them in the PR description:
   
   #1746
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI changes:
   This API integrates our different rebalance tools and serves to provide a single access point that returns optimal replica assignments of a current cluster or given future potential change. 
   
   ### Tests
   
   - [X] The following tests are written for this issue:
   
   TestResourceAssignmentOptimizer.testComputePartitionAssignment
   TestResourceAssignmentOptimizer.testComputePartitionAssignmentWaged
   TestResourceAssignmentOptimizer.testComputePartitionAssignmentNegativeInput
   
   - The following is the result of the "mvn test" command on the appropriate module:
   
   (If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)
   
   ### Changes that Break Backward Compatibility (Optional)
   
   - My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:
   
   (Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)
   
   ### Documentation (Optional)
   
   - In case of new functionality, my PR adds documentation in the following wiki page:
   
   (Link the GitHub wiki you added)
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644364739



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {

Review comment:
       TFTR. Updated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on pull request #1747:
URL: https://github.com/apache/helix/pull/1747#issuecomment-854991883


   Thanks all for reviewing this PR! Rebased to TOT master and waiting for CI to complete.
   
   This PR is ready to be merged.  Approved by @jiajunwang 
   Final commit message:
   New REST api partitionAssignment -- return potential assignment given cluster change
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] junkaixue merged pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
junkaixue merged pull request #1747:
URL: https://github.com/apache/helix/pull/1747


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644314658



##########
File path: helix-rest/src/test/java/org/apache/helix/rest/server/AbstractTestClass.java
##########
@@ -316,7 +316,7 @@ private void setupZooKeepers() {
   }
 
   protected void setupHelixResources() {
-    _clusters = createClusters(3);
+    _clusters = createClusters(4);

Review comment:
       I agree that the optimal way is for each test to create cluster inside the test itself. However, this need a larger code refactor for all our helix-rest integration test. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r643546075



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -201,6 +201,10 @@ public ZkClient getZkClient() {
     return (ZkClient) getRealmAwareZkClient();
   }
 
+  public String getZkAddr() {

Review comment:
       TFTR. Updated with newly added API in rebalancer tool.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r643561023



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();

Review comment:
       Filter fields will need to return if any given participant/resource is in this allowed list. So `Set` data structure is used here. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r643546653



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {

Review comment:
       TFTR. Updated. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r638252430



##########
File path: helix-rest/src/test/java/org/apache/helix/rest/server/TestResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,201 @@
+package org.apache.helix.rest.server;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.ws.rs.client.Entity;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.TestHelper;
+import org.apache.helix.manager.zk.ZKHelixDataAccessor;
+import org.apache.helix.model.IdealState;
+import org.testng.Assert;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+public class TestResourceAssignmentOptimizer extends AbstractTestClass {
+
+  String cluster = "TestCluster_2";
+  String instance1 = cluster + "dummyInstance_localhost_12930";
+  String swapNewInstance = "swapNewInstance";
+  String urlBase = "clusters/TestCluster_2/partitionAssignment/";
+  String swapOldInstance, toRemoveInstance;
+  HelixDataAccessor helixDataAccessor;
+  List<String> resources;
+  List<String> liveInstances;
+
+  @BeforeClass
+  public void beforeClass() {
+    helixDataAccessor = new ZKHelixDataAccessor(cluster, _baseAccessor);
+    _gSetupTool.addInstanceToCluster(cluster, instance1);
+    resources = _gSetupTool.getClusterManagementTool().getResourcesInCluster(cluster);
+    liveInstances =  helixDataAccessor.getChildNames(helixDataAccessor.keyBuilder().liveInstances());
+    Assert.assertFalse(resources.isEmpty() || liveInstances.isEmpty());
+
+    toRemoveInstance = liveInstances.get(0);
+    swapOldInstance = liveInstances.get(1);
+
+  }
+
+  @Test
+  public void testComputePartitionAssignment() throws IOException {
+    System.out.println("Start test :" + TestHelper.getTestMethodName());
+
+    // set all resource to FULL_AUTO except one
+    for (int i = 0; i < resources.size() - 1; ++i) {
+      String resource = resources.get(i);
+      IdealState idealState =
+          _gSetupTool.getClusterManagementTool().getResourceIdealState(cluster, resource);
+      idealState.setRebalanceMode(IdealState.RebalanceMode.FULL_AUTO);

Review comment:
       By default, all resources are created SEMI_AUTO. We kept one resource SEMI_AUTO and set the rest to FULL_AUTO. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644311645



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +281,18 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
     return _byteArrayZkBaseDataAccessor;
   }
 
+  public ZkBucketDataAccessor getZkBucketDataAccessor() {
+    if (_zkBucketDataAccessor == null) {
+      synchronized (this) {
+        if (_zkBucketDataAccessor == null) {
+          _zkBucketDataAccessor = new ZkBucketDataAccessor(_zkAddr);

Review comment:
       ZkBucketDataAccessor will handle the multi-ZK configuration. This is a workaround since we can not create a zkClient from a zkClient.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r643576324



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {

Review comment:
       If you define a class with the field names like INSTANCE_CHANGE, then ObjectMapper can handle it automatically for you. It would help to simplify code, and more importantly, reduce bug and maintenance costs.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +281,18 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
     return _byteArrayZkBaseDataAccessor;
   }
 
+  public ZkBucketDataAccessor getZkBucketDataAccessor() {
+    if (_zkBucketDataAccessor == null) {
+      synchronized (this) {
+        if (_zkBucketDataAccessor == null) {
+          _zkBucketDataAccessor = new ZkBucketDataAccessor(_zkAddr);

Review comment:
       This still looks not right. Any client that is created without using createRealmAwareZkClient() is not compatible with the multi-ZK configuration, right?

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_REMOVE_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_SWAP_INSTANCES) && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case OPTIONS:
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals(OPTIONS_RESOURCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals(OPTIONS_INSTANCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)

Review comment:
       Have you explored the possibility of keeping a cluster data cache here? So we avoid read latency and we don't need to manually reconstruct a simplified rebalance logic here.
   
   Let's add a TODO here at least.

##########
File path: helix-rest/src/test/java/org/apache/helix/rest/server/AbstractTestClass.java
##########
@@ -316,7 +316,7 @@ private void setupZooKeepers() {
   }
 
   protected void setupHelixResources() {
-    _clusters = createClusters(3);
+    _clusters = createClusters(4);

Review comment:
       Just curious, why we need to modify the cluster count when adding new tests? Can't we create the cluster inside the new test cases? Or are we supposed to modify more than one test file every time when we need to add a test case?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644333768



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_REMOVE_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_SWAP_INSTANCES) && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case OPTIONS:
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals(OPTIONS_RESOURCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals(OPTIONS_INSTANCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instances and new instances from user input to instances list.
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance.
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,

Review comment:
       HelixUtil already have multiple different APIs serving this feature. Also `InputFields` and `ClusterState` are private class and only have meaning in this context. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] zhangmeng916 commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
zhangmeng916 commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r637432144



##########
File path: helix-rest/src/test/java/org/apache/helix/rest/server/TestResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,201 @@
+package org.apache.helix.rest.server;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.ws.rs.client.Entity;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.TestHelper;
+import org.apache.helix.manager.zk.ZKHelixDataAccessor;
+import org.apache.helix.model.IdealState;
+import org.testng.Assert;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+public class TestResourceAssignmentOptimizer extends AbstractTestClass {
+
+  String cluster = "TestCluster_2";
+  String instance1 = cluster + "dummyInstance_localhost_12930";
+  String swapNewInstance = "swapNewInstance";
+  String urlBase = "clusters/TestCluster_2/partitionAssignment/";
+  String swapOldInstance, toRemoveInstance;
+  HelixDataAccessor helixDataAccessor;
+  List<String> resources;
+  List<String> liveInstances;
+
+  @BeforeClass
+  public void beforeClass() {
+    helixDataAccessor = new ZKHelixDataAccessor(cluster, _baseAccessor);
+    _gSetupTool.addInstanceToCluster(cluster, instance1);
+    resources = _gSetupTool.getClusterManagementTool().getResourcesInCluster(cluster);
+    liveInstances =  helixDataAccessor.getChildNames(helixDataAccessor.keyBuilder().liveInstances());
+    Assert.assertFalse(resources.isEmpty() || liveInstances.isEmpty());
+
+    toRemoveInstance = liveInstances.get(0);
+    swapOldInstance = liveInstances.get(1);
+
+  }
+
+  @Test
+  public void testComputePartitionAssignment() throws IOException {
+    System.out.println("Start test :" + TestHelper.getTestMethodName());
+
+    // set all resource to FULL_AUTO except one
+    for (int i = 0; i < resources.size() - 1; ++i) {
+      String resource = resources.get(i);
+      IdealState idealState =
+          _gSetupTool.getClusterManagementTool().getResourceIdealState(cluster, resource);
+      idealState.setRebalanceMode(IdealState.RebalanceMode.FULL_AUTO);

Review comment:
       SEMI_AUTO case is not tested. Shall we also test that case?

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());

Review comment:
       We should log this class's name here.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":

Review comment:
       I'd suggest to put the acceptable commands in an enum so that it is easy to track and validate. You can check the `Command` enum in `AbstractResource`

##########
File path: helix-rest/src/test/java/org/apache/helix/rest/server/TestResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,201 @@
+package org.apache.helix.rest.server;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.ws.rs.client.Entity;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.TestHelper;
+import org.apache.helix.manager.zk.ZKHelixDataAccessor;
+import org.apache.helix.model.IdealState;
+import org.testng.Assert;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+public class TestResourceAssignmentOptimizer extends AbstractTestClass {
+
+  String cluster = "TestCluster_2";
+  String instance1 = cluster + "dummyInstance_localhost_12930";
+  String swapNewInstance = "swapNewInstance";
+  String urlBase = "clusters/TestCluster_2/partitionAssignment/";
+  String swapOldInstance, toRemoveInstance;
+  HelixDataAccessor helixDataAccessor;
+  List<String> resources;
+  List<String> liveInstances;
+
+  @BeforeClass
+  public void beforeClass() {
+    helixDataAccessor = new ZKHelixDataAccessor(cluster, _baseAccessor);
+    _gSetupTool.addInstanceToCluster(cluster, instance1);
+    resources = _gSetupTool.getClusterManagementTool().getResourcesInCluster(cluster);
+    liveInstances =  helixDataAccessor.getChildNames(helixDataAccessor.keyBuilder().liveInstances());
+    Assert.assertFalse(resources.isEmpty() || liveInstances.isEmpty());
+
+    toRemoveInstance = liveInstances.get(0);
+    swapOldInstance = liveInstances.get(1);
+

Review comment:
       NIT: There're some blank lines in some files, and some typos in comments. Maybe you can double check and reformat.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();

Review comment:
       Is there a reason that some fields are list, while others are set? Keep them consistent is better.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals("AddInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("RemoveInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("SwapInstances") && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case "Options":
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals("ResourceFilter") && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals("InstanceFilter") && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command for invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instance and new instance from user input to instances list
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,
+      ClusterState clusterState, String clusterId) throws Exception {
+
+    AssignmentResult result = new AssignmentResult();
+    // Iterate through resources, read resource level info and get potential assignment.
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    List<IdealState> wagedResourceIdealState = new ArrayList<>();
+
+    for (String resource : clusterState.resources) {
+      IdealState idealState =
+          dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(resource));
+      // Compute all Waged resources in a batch later.
+      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
+          .equals(WagedRebalancer.class.getName())) {
+        wagedResourceIdealState.add(idealState);
+        continue;
+      }
+      // For non Waged resources, we don't compute resources not in white list.
+      if (!inputFields.resourceFilter.isEmpty() && !inputFields.resourceFilter.contains(resource)) {
+        continue;
+      }
+      // Use getIdealAssignmentForFullAuto for FULL_AUTO resource.
+      if (idealState.getRebalanceMode() == IdealState.RebalanceMode.FULL_AUTO) {
+        String rebalanceStrategy = idealState.getRebalanceStrategy();
+        if (rebalanceStrategy == null || rebalanceStrategy
+            .equalsIgnoreCase(RebalanceStrategy.DEFAULT_REBALANCE_STRATEGY)) {
+          rebalanceStrategy = AutoRebalanceStrategy.class.getName();
+        }
+        Map<String, Map<String, String>> partitionAssignments = new TreeMap<>(
+            getIdealAssignmentForFullAuto(clusterState.clusterConfig, clusterState.instanceConfigs,
+                clusterState.instances, idealState, new ArrayList<>(idealState.getPartitionSet()),
+                rebalanceStrategy));
+        instanceSwapAndFilter(inputFields, partitionAssignments, resource, result);

Review comment:
       Also this function can be called once instead of three times as it's used by each case.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals("AddInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("RemoveInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("SwapInstances") && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case "Options":
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals("ResourceFilter") && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals("InstanceFilter") && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command for invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instance and new instance from user input to instances list
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,
+      ClusterState clusterState, String clusterId) throws Exception {
+
+    AssignmentResult result = new AssignmentResult();
+    // Iterate through resources, read resource level info and get potential assignment.
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    List<IdealState> wagedResourceIdealState = new ArrayList<>();
+
+    for (String resource : clusterState.resources) {
+      IdealState idealState =
+          dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(resource));
+      // Compute all Waged resources in a batch later.
+      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
+          .equals(WagedRebalancer.class.getName())) {
+        wagedResourceIdealState.add(idealState);
+        continue;
+      }
+      // For non Waged resources, we don't compute resources not in white list.
+      if (!inputFields.resourceFilter.isEmpty() && !inputFields.resourceFilter.contains(resource)) {
+        continue;
+      }
+      // Use getIdealAssignmentForFullAuto for FULL_AUTO resource.
+      if (idealState.getRebalanceMode() == IdealState.RebalanceMode.FULL_AUTO) {

Review comment:
       It's worth to have private functions to implement FULL_AUTO and SEMI_AUTO scenarios. Then this compute function would be much clear and organized.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {

Review comment:
       For the class naming, I think we can follow the convention for rest class, i.e. "***accessor" so that we can easily identify it is rest.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644364646



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals("AddInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("RemoveInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("SwapInstances") && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case "Options":
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals("ResourceFilter") && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals("InstanceFilter") && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command for invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instance and new instance from user input to instances list
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,
+      ClusterState clusterState, String clusterId) throws Exception {
+
+    AssignmentResult result = new AssignmentResult();
+    // Iterate through resources, read resource level info and get potential assignment.
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    List<IdealState> wagedResourceIdealState = new ArrayList<>();
+
+    for (String resource : clusterState.resources) {
+      IdealState idealState =
+          dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(resource));
+      // Compute all Waged resources in a batch later.
+      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
+          .equals(WagedRebalancer.class.getName())) {
+        wagedResourceIdealState.add(idealState);
+        continue;
+      }
+      // For non Waged resources, we don't compute resources not in white list.
+      if (!inputFields.resourceFilter.isEmpty() && !inputFields.resourceFilter.contains(resource)) {
+        continue;
+      }
+      // Use getIdealAssignmentForFullAuto for FULL_AUTO resource.
+      if (idealState.getRebalanceMode() == IdealState.RebalanceMode.FULL_AUTO) {
+        String rebalanceStrategy = idealState.getRebalanceStrategy();
+        if (rebalanceStrategy == null || rebalanceStrategy
+            .equalsIgnoreCase(RebalanceStrategy.DEFAULT_REBALANCE_STRATEGY)) {
+          rebalanceStrategy = AutoRebalanceStrategy.class.getName();
+        }
+        Map<String, Map<String, String>> partitionAssignments = new TreeMap<>(
+            getIdealAssignmentForFullAuto(clusterState.clusterConfig, clusterState.instanceConfigs,
+                clusterState.instances, idealState, new ArrayList<>(idealState.getPartitionSet()),
+                rebalanceStrategy));
+        instanceSwapAndFilter(inputFields, partitionAssignments, resource, result);

Review comment:
       `instanceSwapAndFilter` is called for each resources. For non Waged resources, we compute assignment one by one and filter result one by one. Waged resources need to be computed in a batch and filter result one by one. 
   I am not sure if these three can be merged. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r643726075



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();

Review comment:
       TFTR. Actually they are not. I will add a TODO for future.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r638221054



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -201,6 +201,10 @@ public ZkClient getZkClient() {
     return (ZkClient) getRealmAwareZkClient();
   }
 
+  public String getZkAddr() {

Review comment:
       I think we don't want to expose the ZkAddr to avoid unnecessary dependencies. In a multi ZK scenario, the address might be a mock address or a fake one. Or we may eventually get rid of this field.
   
   This method is the only safe option,   public RealmAwareZkClient getRealmAwareZkClient(); However, there is no way to get "address" from a RealmAwareZkClient interface.
   Although this is a problem that we have noticed for a long time, it seems that we are still facing the same problem.
   
   IMO, the right solution is to keep refactoring the WAGED tool even WAGED rebalancer so it accepts RealmAwareZkClient instead of the raw address. This means we ensure our assumption, the physical ZK address is hidden in the RealmAwareZkClient interface, is guaranteed through all of our code.
   
   However, I'm not sure how much will this change cost. It seems to be a must.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;

Review comment:
       Please avoid static imports like this. HelixUtil would be a good reference for the code readers to understand where the methods are coming from.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":

Review comment:
       All the keys, commands should be enum or static Strings whenever possible. Hard code String is not good.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644357771



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals("AddInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("RemoveInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("SwapInstances") && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case "Options":
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals("ResourceFilter") && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals("InstanceFilter") && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command for invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instance and new instance from user input to instances list
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,
+      ClusterState clusterState, String clusterId) throws Exception {
+
+    AssignmentResult result = new AssignmentResult();
+    // Iterate through resources, read resource level info and get potential assignment.
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    List<IdealState> wagedResourceIdealState = new ArrayList<>();
+
+    for (String resource : clusterState.resources) {
+      IdealState idealState =
+          dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(resource));
+      // Compute all Waged resources in a batch later.
+      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
+          .equals(WagedRebalancer.class.getName())) {
+        wagedResourceIdealState.add(idealState);
+        continue;
+      }
+      // For non Waged resources, we don't compute resources not in white list.
+      if (!inputFields.resourceFilter.isEmpty() && !inputFields.resourceFilter.contains(resource)) {
+        continue;
+      }
+      // Use getIdealAssignmentForFullAuto for FULL_AUTO resource.
+      if (idealState.getRebalanceMode() == IdealState.RebalanceMode.FULL_AUTO) {

Review comment:
       Thanks for suggestion. I have tried and actually I am not sure if adding a separate function for several lines with many parameters would be more organized...




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] junkaixue commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
junkaixue commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r643707579



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();

Review comment:
       Are these cached? If yes, better cached with some objected instead of static fields. Because the data cached is not synchronized between requests. Shared simple data structure could be problematic.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +281,18 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
     return _byteArrayZkBaseDataAccessor;
   }
 
+  public ZkBucketDataAccessor getZkBucketDataAccessor() {
+    if (_zkBucketDataAccessor == null) {
+      synchronized (this) {
+        if (_zkBucketDataAccessor == null) {
+          _zkBucketDataAccessor = new ZkBucketDataAccessor(_zkAddr);

Review comment:
       +1. I was trying to make this comment, for realm aware client, get a dedicated zk address client is not a good idea.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange

Review comment:
       Can we have nested switch case for here? That could be easy for us to read.

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_REMOVE_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_SWAP_INSTANCES) && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case OPTIONS:
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals(OPTIONS_RESOURCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals(OPTIONS_INSTANCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instances and new instances from user input to instances list.
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance.
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,

Review comment:
       Shall we move the logic into HelixUtil or RebalanceUtil? Because this could be something leveraged by Java API as well.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
xyuanlu commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r644311645



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/ServerContext.java
##########
@@ -278,6 +281,18 @@ public HelixDataAccessor getDataAccessor(String clusterName) {
     return _byteArrayZkBaseDataAccessor;
   }
 
+  public ZkBucketDataAccessor getZkBucketDataAccessor() {
+    if (_zkBucketDataAccessor == null) {
+      synchronized (this) {
+        if (_zkBucketDataAccessor == null) {
+          _zkBucketDataAccessor = new ZkBucketDataAccessor(_zkAddr);

Review comment:
       ZkBucketDataAccessor will handle the multi-ZK configuration. This is a workaround since we can not create a zkClient from a zkClient.

##########
File path: helix-rest/src/test/java/org/apache/helix/rest/server/AbstractTestClass.java
##########
@@ -316,7 +316,7 @@ private void setupZooKeepers() {
   }
 
   protected void setupHelixResources() {
-    _clusters = createClusters(3);
+    _clusters = createClusters(4);

Review comment:
       I agree that the optimal way is for each test to create cluster inside the test itself. However, this need a larger code refactor for all our helix-rest integration test. 

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_REMOVE_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_SWAP_INSTANCES) && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case OPTIONS:
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals(OPTIONS_RESOURCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals(OPTIONS_INSTANCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instances and new instances from user input to instances list.
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance.
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,

Review comment:
       HelixUtil already have multiple different APIs serving this feature. Also `InputFields` and `ClusterState` are private class and only have meaning in this context. 

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals("AddInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("RemoveInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("SwapInstances") && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case "Options":
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals("ResourceFilter") && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals("InstanceFilter") && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command for invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instance and new instance from user input to instances list
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,
+      ClusterState clusterState, String clusterId) throws Exception {
+
+    AssignmentResult result = new AssignmentResult();
+    // Iterate through resources, read resource level info and get potential assignment.
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    List<IdealState> wagedResourceIdealState = new ArrayList<>();
+
+    for (String resource : clusterState.resources) {
+      IdealState idealState =
+          dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(resource));
+      // Compute all Waged resources in a batch later.
+      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
+          .equals(WagedRebalancer.class.getName())) {
+        wagedResourceIdealState.add(idealState);
+        continue;
+      }
+      // For non Waged resources, we don't compute resources not in white list.
+      if (!inputFields.resourceFilter.isEmpty() && !inputFields.resourceFilter.contains(resource)) {
+        continue;
+      }
+      // Use getIdealAssignmentForFullAuto for FULL_AUTO resource.
+      if (idealState.getRebalanceMode() == IdealState.RebalanceMode.FULL_AUTO) {

Review comment:
       Thanks for suggestion. I have tried and actually I am not sure if adding a separate function for several lines with many parameters would be more organized...

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizer.java
##########
@@ -0,0 +1,347 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.helix.util.HelixUtil.computeIdealMapping;
+import static org.apache.helix.util.HelixUtil.getIdealAssignmentForFullAuto;
+import static org.apache.helix.util.HelixUtil.getTargetAssignmentForWagedFullAuto;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizer extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory
+      .getLogger(org.apache.helix.rest.server.resources.helix.ClusterAccessor.class.getName());
+
+  private static class InputFields {
+    Set<String> resourceFilter = new HashSet<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Set<String> instanceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV mapping. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can but be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error(
+          "OutOfMemoryError while calling AssignmentResult" + Arrays.toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case "InstanceChange":
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals("AddInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("RemoveInstances") && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals("SwapInstances") && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case "Options":
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals("ResourceFilter") && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals("InstanceFilter") && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command for invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instance and new instance from user input to instances list
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,
+      ClusterState clusterState, String clusterId) throws Exception {
+
+    AssignmentResult result = new AssignmentResult();
+    // Iterate through resources, read resource level info and get potential assignment.
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    List<IdealState> wagedResourceIdealState = new ArrayList<>();
+
+    for (String resource : clusterState.resources) {
+      IdealState idealState =
+          dataAccessor.getProperty(dataAccessor.keyBuilder().idealStates(resource));
+      // Compute all Waged resources in a batch later.
+      if (idealState.getRebalancerClassName() != null && idealState.getRebalancerClassName()
+          .equals(WagedRebalancer.class.getName())) {
+        wagedResourceIdealState.add(idealState);
+        continue;
+      }
+      // For non Waged resources, we don't compute resources not in white list.
+      if (!inputFields.resourceFilter.isEmpty() && !inputFields.resourceFilter.contains(resource)) {
+        continue;
+      }
+      // Use getIdealAssignmentForFullAuto for FULL_AUTO resource.
+      if (idealState.getRebalanceMode() == IdealState.RebalanceMode.FULL_AUTO) {
+        String rebalanceStrategy = idealState.getRebalanceStrategy();
+        if (rebalanceStrategy == null || rebalanceStrategy
+            .equalsIgnoreCase(RebalanceStrategy.DEFAULT_REBALANCE_STRATEGY)) {
+          rebalanceStrategy = AutoRebalanceStrategy.class.getName();
+        }
+        Map<String, Map<String, String>> partitionAssignments = new TreeMap<>(
+            getIdealAssignmentForFullAuto(clusterState.clusterConfig, clusterState.instanceConfigs,
+                clusterState.instances, idealState, new ArrayList<>(idealState.getPartitionSet()),
+                rebalanceStrategy));
+        instanceSwapAndFilter(inputFields, partitionAssignments, resource, result);

Review comment:
       `instanceSwapAndFilter` is called for each resources. For non Waged resources, we compute assignment one by one and filter result one by one. Waged resources need to be computed in a batch and filter result one by one. 
   I am not sure if these three can be merged. 

##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {

Review comment:
       TFTR. Updated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r645838687



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_REMOVE_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_SWAP_INSTANCES) && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case OPTIONS:
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals(OPTIONS_RESOURCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals(OPTIONS_INSTANCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instances and new instances from user input to instances list.
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance.
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,

Review comment:
       @junkaixue This logic (together with the other compute methods) is needed only when we don't have the cluster data cache maintained for helix-rest. I believe adding cluster data cache in helix-rest is our next step. And after that, we can just run a rebalance pipeline instead of duplicating these logics here.
   As I discussed with Xiaoyuan, the next step will take time. So we proposed to have this private method temporarily. But we should keep it private since it will be removed eventually.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] jiajunwang commented on a change in pull request #1747: New REST api partitionAssignment -- return potential assignment given cluster change

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1747:
URL: https://github.com/apache/helix/pull/1747#discussion_r645838687



##########
File path: helix-rest/src/main/java/org/apache/helix/rest/server/resources/helix/ResourceAssignmentOptimizerAccessor.java
##########
@@ -0,0 +1,354 @@
+package org.apache.helix.rest.server.resources.helix;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.security.InvalidParameterException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.stream.Collectors;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+
+import com.codahale.metrics.annotation.ResponseMetered;
+import com.codahale.metrics.annotation.Timed;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import org.apache.helix.ConfigAccessor;
+import org.apache.helix.HelixDataAccessor;
+import org.apache.helix.controller.rebalancer.strategy.AutoRebalanceStrategy;
+import org.apache.helix.controller.rebalancer.strategy.RebalanceStrategy;
+import org.apache.helix.controller.rebalancer.waged.WagedRebalancer;
+import org.apache.helix.manager.zk.ZkBaseDataAccessor;
+import org.apache.helix.model.ClusterConfig;
+import org.apache.helix.model.IdealState;
+import org.apache.helix.model.InstanceConfig;
+import org.apache.helix.model.ResourceAssignment;
+import org.apache.helix.model.ResourceConfig;
+import org.apache.helix.model.StateModelDefinition;
+import org.apache.helix.rest.common.HttpConstants;
+import org.apache.helix.util.HelixUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+@Path("/clusters/{clusterId}/partitionAssignment")
+public class ResourceAssignmentOptimizerAccessor extends AbstractHelixResource {
+  private static Logger LOG = LoggerFactory.getLogger(
+      org.apache.helix.rest.server.resources.helix.ResourceAssignmentOptimizerAccessor.class
+          .getName());
+
+  private static class InputFields {
+    List<String> newInstances = new ArrayList<>();
+    List<String> instancesToRemove = new ArrayList<>();
+    Map<String, String> nodeSwap = new HashMap<>(); // old instance -> new instance.
+    Set<String> instanceFilter = new HashSet<>();
+    Set<String> resourceFilter = new HashSet<>();
+  }
+
+  private static class ClusterState {
+    List<InstanceConfig> instanceConfigs = new ArrayList<>();
+    ClusterConfig clusterConfig;
+    List<String> resources = new ArrayList<>();
+    List<String> instances;        // cluster LiveInstance + addInstances - instancesToRemove.
+  }
+
+  // Result format: Map of resource -> partition -> instance -> state.
+  private static class AssignmentResult extends HashMap<String, Map<String, Map<String, String>>> {
+    public AssignmentResult() {
+      super();
+    }
+  }
+
+  public static final String INSTANCE_CHANGE = "InstanceChange";
+  public static final String INSTANCE_CHANGE_ADD_INSTANCES = "AddInstances";
+  public static final String INSTANCE_CHANGE_REMOVE_INSTANCES = "RemoveInstances";
+  public static final String INSTANCE_CHANGE_SWAP_INSTANCES = "SwapInstances";
+  public static final String OPTIONS = "Options";
+  public static final String OPTIONS_INSTANCE_FLT = "InstanceFilter";
+  public static final String OPTIONS_RESOURCE_FLT = "ResourceFilter";
+
+  @ResponseMetered(name = HttpConstants.WRITE_REQUEST)
+  @Timed(name = HttpConstants.WRITE_REQUEST)
+  @POST
+  public Response computePotentialAssignment(@PathParam("clusterId") String clusterId,
+      String content) {
+
+    InputFields inputFields;
+    ClusterState clusterState;
+    AssignmentResult result;
+
+    try {
+      // 1.  Try to parse the content string. If parseable, use it as a KV map. Otherwise, return a REASON String
+      inputFields = readInput(content);
+      // 2. Read cluster status from ZK.
+      clusterState = readClusterStateAndValidateInput(clusterId, inputFields);
+      // 3. Call rebalancer tools for each resource.
+      result = computeOptimalAssignmentForResources(inputFields, clusterState, clusterId);
+      // 4. Serialize result to JSON and return.
+      return JSONRepresentation(result);
+    } catch (InvalidParameterException ex) {
+      return badRequest(ex.getMessage());
+    } catch (JsonProcessingException e) {
+      return badRequest("Invalid input: Input can not be parsed into a KV map." + e.getMessage());
+    } catch (OutOfMemoryError e) {
+      LOG.error("OutOfMemoryError while calling partitionAssignment" + Arrays
+          .toString(e.getStackTrace()));
+      return badRequest(
+          "Response size is too large to serialize. Please query by resources or instance filter");
+    } catch (Exception e) {
+      LOG.error("Failed to compute partition assignment:" + Arrays.toString(e.getStackTrace()));
+      return badRequest("Failed to compute partition assignment: " + e);
+    }
+  }
+
+  private InputFields readInput(String content)
+      throws InvalidParameterException, JsonProcessingException {
+
+    Map<String, Map<String, Object>> customFieldsMap;
+    InputFields inputMap = new InputFields();
+    customFieldsMap =
+        OBJECT_MAPPER.readValue(content, new TypeReference<HashMap<String, Map<String, Object>>>() {
+        });
+
+    // Content is given as a KV mapping.
+    for (Map.Entry<String, Map<String, Object>> entry : customFieldsMap.entrySet()) {
+      String key = entry.getKey();
+      switch (key) {
+        case INSTANCE_CHANGE:
+          for (Map.Entry<String, Object> instanceChange : entry.getValue().entrySet()) {
+            String instanceChangeKey = instanceChange.getKey();
+            if (instanceChangeKey.equals(INSTANCE_CHANGE_ADD_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.newInstances.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_REMOVE_INSTANCES) && (instanceChange
+                .getValue() instanceof List)) {
+              inputMap.instancesToRemove.addAll((List<String>) instanceChange.getValue());
+            } else if (instanceChangeKey.equals(INSTANCE_CHANGE_SWAP_INSTANCES) && (instanceChange
+                .getValue() instanceof Map)) {
+              for (Map.Entry<String, String> swapPair : ((Map<String, String>) instanceChange
+                  .getValue()).entrySet()) {
+                inputMap.nodeSwap.put(swapPair.getKey(), swapPair.getValue());
+              }
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for InstanceChange : "
+                      + instanceChangeKey);
+            }
+          }
+          break;
+        case OPTIONS:
+          for (Map.Entry<String, Object> option : entry.getValue().entrySet()) {
+            String optionKey = option.getKey();
+            if (optionKey.equals(OPTIONS_RESOURCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.resourceFilter.addAll((List<String>) option.getValue());
+            } else if (optionKey.equals(OPTIONS_INSTANCE_FLT) && (option.getValue() instanceof List)) {
+              inputMap.instanceFilter.addAll((List<String>) option.getValue());
+            } else {
+              throw new InvalidParameterException(
+                  "Unsupported command or invalid format for Options : " + option);
+            }
+          }
+          break;
+        default:
+          throw new InvalidParameterException(
+              "Unsupported command for partitionAssignment : " + key);
+      }
+    }
+    return inputMap;
+  }
+
+  private ClusterState readClusterStateAndValidateInput(String clusterId, InputFields inputFields)
+      throws InvalidParameterException {
+
+    ClusterState clusterState = new ClusterState();
+    ConfigAccessor cfgAccessor = getConfigAccessor();
+    HelixDataAccessor dataAccessor = getDataAccssor(clusterId);
+    clusterState.resources = dataAccessor.getChildNames(dataAccessor.keyBuilder().idealStates());
+    // Add existing live instances and new instances from user input to instances list.
+    clusterState.instances = dataAccessor.getChildNames(dataAccessor.keyBuilder().liveInstances());
+    clusterState.instances.addAll(inputFields.newInstances);
+
+    // Check if to be removed instances and old instances in swap node exist in live instance.
+    if (!inputFields.nodeSwap.isEmpty() || !inputFields.instancesToRemove.isEmpty()) {
+      Set<String> liveInstanceSet = new HashSet<>(clusterState.instances);
+      for (Map.Entry<String, String> nodeSwapPair : inputFields.nodeSwap.entrySet()) {
+        if (!liveInstanceSet.contains(nodeSwapPair.getKey())) {
+          throw new InvalidParameterException("Invalid input: instance [" + nodeSwapPair.getKey()
+              + "] in SwapInstances does not exist in cluster.");
+        }
+      }
+      for (String instanceToRemove : inputFields.instancesToRemove) {
+        if (!liveInstanceSet.contains(instanceToRemove)) {
+          throw new InvalidParameterException("Invalid input: instance [" + instanceToRemove
+              + "] in RemoveInstances does not exist in cluster.");
+        }
+      }
+      if (!inputFields.instancesToRemove.isEmpty()) {
+        clusterState.instances.removeIf(inputFields.instancesToRemove::contains);
+      }
+    }
+
+    // Read instance and cluster config.
+    // It will throw exception if there is no instanceConfig for newly added instance.
+    for (String instance : clusterState.instances) {
+      InstanceConfig config = cfgAccessor.getInstanceConfig(clusterId, instance);
+      clusterState.instanceConfigs.add(config);
+    }
+    clusterState.clusterConfig = cfgAccessor.getClusterConfig(clusterId);
+    return clusterState;
+  }
+
+  private AssignmentResult computeOptimalAssignmentForResources(InputFields inputFields,

Review comment:
       @junkaixue This logic is needed only when we don't have the cluster data cache maintained for helix-rest. I believe adding cluster data cache in helix-rest is our next step. And after that, we can just run a rebalance pipeline instead of duplicating these logics here.
   As I discussed with Xiaoyuan, the next step will take time. So we proposed to have this private method temporarily. But we should keep it private since it will be removed eventually.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org